I figured it out, I had my configuration set to base tax on store location instead of shipping address.