Hmmm ... Zen Cart already sends the application/octet-stream notation. But, it uses a capital "A".
Would you mind doing a quick test?
Edit /includes/modules/pages/download/header_php.php and change the line (I think it's line 121) that says this:
Code:
header("Content-Type: Application/octet-stream");
to use a lower-case "a", like this:
Code:
header("Content-Type: application/octet-stream");
... and temporarily remove your .htaccess change, and see if the .rar downloads properly without the need for the .htaccess edit?
I'd be interested in the results you get.