This USPS mod will let you do that
USPS Shipping Module Update - RateV4 - IntlRateV2
It has a "Filter by Product Volume" or "Filter by Weight" function built in.
Here is an example of how we use it which sounds similar to your needs:
A small flatrate box will hold 67 cubic inches. We have a product that which we can fit a quantity of 5 in a small flatrat box. So, we use 13 cubic inches as the volume of that product. If a customer has 5 or less of that product in the cart, then the small flatrate box is an available option. If a 6th is added, then the small flat rate box is no longer an option and the customer is required to use a Large Video Mailer box, Regional Flat Rate A or a Medium flat rate box. More than 8 will not fit into a Large Video mailer box, so it is automatically removed from the available options when the 9th item is added to the cart leaving the Regional FlatRate A box and the Medium FlatRate box.
We have based the volume of other items that we sell to cause the smaller boxes to be removed as necessary.
We also do not wish to ship the product described by firstclass mail, so we modified the coding so that 12.5 cubic inches is the max that we will ship firstclass. If the example above is added to the cart, the firstclass option is automatically removed but other items of lesser volume are still allowed the firstclass option.
The module does require a SQL patch to be applied to the database so that digits 6-11 in the weight field can be utilized for the product volume. That's listed in the instructions.
Rick
RixStix (dot) com
aka: ChainWeavers (dot) com