So after some tinkering I figured out that the only way to enable Stripe as the only payment method is to have COD installed but disabled by setting "do you wish to accept COD payments" to False.
Another way, through the back end, was to update <store prefix>_configuration table.configuration_value.
There should not be any dependency of Stripe on COD. I am BAU but this issue should be looked into.