Figured it out myself finally! I work on this for quite a while (I won't admit how long) and as soon as I post a message I figure it out 5 minutes later.

I had the item set as a Special. So I needed to set the 'Apply Discounts Used by Product Special/Sale:' to No.

Sorry about that, but hopefully it will help someone else if they do the same thing.