You would really need to customize a shipping module to do this sort of calculations as you cannot use total items in the cart in the calculation as total items is incorrect ...

You cannot use the Flat Rate alone, as that is incorrect ...

What you need is to know how many are from Category B * 5.00 and if there is at least 1 Product from Category A then add 5.00 ...

There are functions in the shopping_cart.php class where you could use the master_categories_id to determine how many Products * Qty come from each Category ...

The results would be:
Products in Categories A >= 1 add 5.00
Products in Categories B add 5.00 * quantity

So a customized shipping module would be the best method to do this ...