You would need to customize both the UPS and Table Rate shipping modules to check the weight and enable or disable them based on weight ...
For example, the Table Rate table shipping module:
/includes/modules/shipping/table.php
Code:
if (zen_get_shipping_enabled($this->code)) {
$this->enabled = ((MODULE_SHIPPING_TABLE_STATUS == 'True') ? true : false);
}
// do NOT show if weight is > 5.00
if (!IS_ADMIN_FLAG) {
global $cart;
if ($_SESSION['cart']->show_weight() > 5.00) {
$this->enabled = false;
}
}
if ( ($this->enabled == true) && ((int)MODULE_SHIPPING_TABLE_ZONE > 0) ) {
Then reverse the logic for the UPS shipping module:
/includes/modules/shipping/ups.php