Limitless:
It will actually be faster, as you won't be dependent on yet another web server serving up the images.
The only time it may help would be if you use a CDN type network, cloudfront, cloudflare, akamai, etc but you would still want to host the images.
The "path-to-image" is nothing more than a REFERENCE in the database, RELATIVE to the CORE PATH defined for a resource's LOCATION.
For "external" images, what you are having to do is OVER-RIDE core behaviour, with CUSTOM behaviour.
CORE behaviour assumes that product images are going to reside in the images folder (or possibly a SUB-folder in the images folder).
The PATH to this folder is established in the configure.php file, and CORE behaviour states that it is whatever-folder-name in the DIR_WS_IMAGES define statement.
Now... once you have made all the edits to the CORE FILES that use "DIR_WS_IMAGES" as the key referencing point for INTERNAL images, you are going to have to MANUALLY ADD the URL to the path for every product image you add.
So... what you need to do is this.
When you ADD a product in your admin console (admin>>>catalog>categories/products>>> add a product), when you get to the part which asks you to add an image, you need to TYPE INTO the field that says:-
Or, select an existing image file from server, filename:
domain.com/images/image1.jpg
WHERE:
domain.com (is the domain of your supplier... do not enter "http://")
/images (is the DIRECTORY on their server where the images live)
/image1.jpg (is the name of the image for the product in question)