You're correct.
If you want your customer to stay on your site and provide their payment details for direct processing, you'll need a secure site. You would need to purchase an SSL certificate for that, or use a shared certificate provided by your hosting company.
This really is the best way.

If you wish your customers' payment details to be collected by a third party, you'd have to use the Connect service or another similar option where the customer leaves your site for payment and returns when payment is complete.
You should still secure your site with an SSL certificate, because you will be storing customers' name/address/phone information and their buying history. This information is technically just as sensitive as payment details, depending on how you look at it.

SSL certificates are not all that expensive; shared certificates may already be offered by your hosting company at nominal or no addditional cost.

If you are needing hosting services designed for eCommerce purposes and are fast, reliable and professional, but don't break the bank, click on the "Certified Hosts" link at the top of this page for some recommendations.