Thanks kuroi,
If I set "not check stock" then customers would be able to buy products that are not in stock? This is not very desirable in my case.

PS: I have just seen the option "Products status in Catalog when out of stock should be set to", changed it to 1, should work now! Thank you for pointing me to admin->config->stock