OK, the idea with separating the options into 2 dropdowns made things even worse...

Here's the admin setup:



Both red and black have price factor 5:



Adding to cart with Black and Red selected, cart shows $100 subtotal.



Going to checkout, subtotal is $0!



Now, if I remove the special price, price factor doesn't work at all on the dropdowns, it just takes the $15 and cart shows $30, checkout subtotal is OK.

There's something REALLY weird going on here...