There are a number of places in the product_listing.php module for the column mod where the code tests for whether the layout should be in rows or columns.
Assuming you have the overall cart set to display columns, and the product category that you want to use rows for has a category_id of XXX, you could change each of the
Code:
if (PRODUCT_LISTING_LAYOUT_STYLE == 'rows')
tests to be
Code:
if (PRODUCT_LISTING_LAYOUT_STYLE == 'rows' OR (int)$current_category_id == XXX)
and each of the
Code:
if (PRODUCT_LISTING_LAYOUT_STYLE == 'columns')
tests to be
Code:
if (PRODUCT_LISTING_LAYOUT_STYLE == 'columns' AND (int)$current_category_id != XXX)