If they choose to go through traditional checkout and log into or create their account on your site, then they can use PayPal as a payment option on the checkout_payment page.
Using this method is no longer "express" in terms of being so-called "faster", but it is still PayPal doing the processing.
If you want to remove the regular checkout button, feel free. It may or may not change the level of confusion for your customers. Only you can decide that.
Bookmarks