Well it turns out the client does want the option of read more when there's an attribute so I had to mix up the stuff.

I still have the Add to Cart and More Info link on items without attributes but now items with atts get a button that says More Info instead of a link. THe links above will represent the final version.

IN order to keep a text link and image link for more info I copied the MORE_INFO_TEXT definition and named it MORE_INFO_TEXT2. Now one is the image version and the other is the text version. Then referenced that accordingly in product_listing.php.