I've never had it slow down my site. The only possible thing I could imagine, and it would be rare, is if it is conflicting with numerous other poorly performing javascript or jquery addons you've added to your admin, in which case you will need to assess all the other addons in your admin for compatibility. You didn't list anything about your site or your addons, so we can't help you with that. You'll have to assess those yourself.

And it would never slow down the storefront part, since it's only installed to the admin side.