Seriously, Serious, you should always accept Paypal regardless of other payment options. Why turn customers away? We have a merchant account, it used to be with First Data which was a huge ripoff but now it's through our bank. Interestingly enough, the backend code is still with First Data and we still use their payment module. But our rates are way, way better.
But even though we do offer credit card payment directly through the site, Paypal is still around 50% of our customer's payment choice. A lot of people use it and are very comfortable with it, and there are plenty of people that don't have credit cards but do have a Paypal account. There are also plenty of people that do have credit cards but trust Paypal with their sensitive card data over you. Believe me, that is very true, we get that regularly. So you need to plan for that 3 to 4 day transfer turnaround, or get a debit card from them so you can use the money in your account to pay vendors, bills, etc. immediately.
Plus there are other benefits. Paypal only charges 30 cents processing fee to refund your customer and refunds the entire amount while a merchant processor is going to charge the % fee for the sale and then the same % fee for a refund and not give you back the original % fee like Paypal does. Believe me, the merchant account companies suck your blood way more than Paypal ever has! They are leeches. Paypal does not distinguish between "qualified" and "non-qualified" cards, which means no rewards points cards at a lower % and rewards points cards at a higher % - we merchants are the ones that fund rewards points you know. So I think you will find they (PayPal) are actually the least $$ solution.
What other problems do you have that makes it tough to keep using them? If you are having other problems with Paypal, like a lot of disputes that tie up your funds, that may be an indicator of other problems with your business model and/or customer service. We get around 500 Paypal transactions a month and have maybe 1 dispute every 2 or 3 months and it almost always - like 95% of the time - resolved amicably through communication.
By all means, get a merchant account and offer credit card payment through your site. You will have to get to a higher level of PCI certification to accept cards directly through your site but that is just part of doing business. But, I strongly recommend against not using Paypal regardless of your other payment methods. You never, ever want to turn customers off from giving you their money! Give them more options, not less.