I have the following scenario that I am trying to find a solution to:

One product has several types (gender?) :
Men's , Women's Toddlers and Baby's ;
Of those, men and women's have the same sizes (L, M, S), but the Toddlers and babies are completely unique - 2, 4, 6, 3month-6month, 6mo.-12mo., 12mo.-18mo. and so on.
So far so good, but then it gets complicated with the colors:
men and women's are purple, slate, blue
but babies and toddlers have only natural and yellow.

My question is: can I somehow make the color attributes dependent on the selection of gender/size, namely, if a customer selects men's size, the color options to be limited to purple, slate, blue. I they select a toddler size, then they would only have natural and yellow colors to select from.

Any suggestions are greatly appreciated.

Thanks.