Symbolic link folder permissions

I'm running ZC 1.3.8a, Linux. My host uses suphp. I have and use SSL (my own, not shared).

I was told that no folders were to have 777 permissions. Instead I am to use 755.

I have been having a variety of "sporadic" download issues over the past month especially with larger files (greater than say 20MB). Downloads have I/O errors, corrupt errors, etc. Webhost has no idea what the problem is and I've researched the forums and haven't found an answer either.

I use download by redirect and have "pub" & "download" folders set to 755. Mostly, I am NOT able to reproduce the download errors that my customers have and neither can my webhost. Customers have all different internet connections, some even DSL. My max_execution_time is set to 900, so I do not suspect any issues with that either. My largest downloads are about 50-60MB and download fine with a direct link.

While doing some testing today, I noticed that the symbolic link folders always have 777 permissions. (Webhost says this should be okay even with suphp.) However, when I made some changes from download by redirect to streaming (redirect=false & streaming=true), I received an "IE cannot display this webpage error" when I clicked on the download link and when I checked my Cpanel error log, it said "Symbolic link not allowed or link target not accessible: /home/.../public_html/store/pub/.vhbgpytsbobpds/...".

Any ideas as to what causes this error?

There are so many variables for me here. I changed to this webhost in July and have had problem after problem with downloads. I came from a freeBSD(?) server with shared SSL and was running ZC1.3.6 using download by redirect. I didn't have all of these download problems then. I'm stumped as to what is causing so many problems now. I have a new cart and what seems to be better host as well as my own SSL - should be in better hands now I'd think.

Any thoughts on where to begin my troubleshooting? I'm also having some trouble with downloads from the CMS side of my site for files as small as 12MB. Maybe that gives me some kind of hint? Webhost insists it's not them.

Debra_in_MO:

I was told that no folders were to have 777 permissions. Instead I am to use 755.With suphp, that's fine.

Debra_in_MO:

Mostly, I am NOT able to reproduce the download errors that my customers have and neither can my webhost. That's puzzling.

Debra_in_MO:

Customers have all different internet connections, some even DSL. My max_execution_time is set to 900, so I do not suspect any issues with that either. My largest downloads are about 50-60MB and download fine with a direct link.With direct or redirected downloads, the max-execution time shouldn't make any difference.
If using download-by-streaming, then the max-execution time does play a part.

Debra_in_MO:

While doing some testing today, I noticed that the symbolic link folders always have 777 permissions. (Webhost says this should be okay even with suphp.) However, when I made some changes from download by redirect to streaming (redirect=false & streaming=true), I received an "IE cannot display this webpage error"If it was truly in streaming mode at that point, that suggests that somehow the streaming isn't starting properly, whether a timeout or a lag in processing, or unexpected page headers are being sent, etc.

Debra_in_MO:

when I clicked on the download link and when I checked my Cpanel error log, it said "Symbolic link not allowed or link target not accessible: /home/.../public_html/store/pub/.vhbgpytsbobpds/...".If you were in streaming mode, you shouldn't be getting any new symbolic link errors, because symlinks aren't used when streaming.
Symlinks are only used in redirect mode.

Debra_in_MO:

I changed to this webhost in July and have had problem after problem with downloads.

... I'm also having some trouble with downloads from the CMS side of my site ... Maybe that gives me some kind of hint? Webhost insists it's not them.
I'm inclined to suggest it's something amuck with the host. :(

I would also agree that the problem probably stems from the Hosting side of things.

The problems stated are not the norm but can happen on an overloaded Server.

You should monitor by using:

Admin > Tools > Server/Version Info

• look at the line that states "Server Up Time" (near top of page) and see what it shows for "load average"

The first set of numbers is most important and hopefully, it will always be under or close to 1.0 but even 3.0 can be acceptable in certain situations. If any higher and on a regular basis, you know the Server is overloaded which will cause all cause of problems.

DrByte:

If you were in streaming mode, you shouldn't be getting any new symbolic link errors, because symlinks aren't used when streaming.
Symlinks are only used in redirect mode.

I'm inclined to suggest it's something amuck with the host. :(

What should a streaming link look like? The link that takes me to the "IE cannot display this webpage" looks like this:
http://www.mydomain.com/store/index.php?main_page=download&order=XXXX&id=XXXXX (X's are numbers)

I tried this also in Firefox and the page just hangs and doesn't do anything.

I have redirect set to false and streaming set to true. Are there other settings I should be changing in order to make the switch from one to the other?

Website Rob:

I would also agree that the problem probably stems from the Hosting side of things.

The problems stated are not the norm but can happen on an overloaded Server.

You should monitor by using:

Admin > Tools > Server/Version Info

• look at the line that states "Server Up Time" (near top of page) and see what it shows for "load average"

The first set of numbers is most important and hopefully, it will always be under or close to 1.0 but even 3.0 can be acceptable in certain situations. If any higher and on a regular basis, you know the Server is overloaded which will cause all cause of problems.

I'll keep an eye on this. Thanks for that info. At first glance the first number was 1.4, but it's steadily decreased throughout the evening. I'll check it during the day tomorrow when things are busier.

Debra_in_MO:

What should a streaming link look like? The link that takes me to the "IE cannot display this webpage" looks like this:
http://www.mydomain.com/store/index.php?main_page=download&order=XXXX&id=XXXXX (X's are numbers)

That's a normal link when in streaming mode.

Debra_in_MO:

I have redirect set to false and streaming set to true. Are there other settings I should be changing in order to make the switch from one to the other?

Not usually.

DrByte:

If it was truly in streaming mode at that point, that suggests that somehow the streaming isn't starting properly, whether a timeout or a lag in processing, or unexpected page headers are being sent, etc.

I assume that "timeout" and "lag in processing" would be something for the webhost to take a look at. What about the "unexpected page headers being sent"? How do I check this?

Website Rob:

I would also agree that the problem probably stems from the Hosting side of things.

The problems stated are not the norm but can happen on an overloaded Server.

You should monitor by using:

Admin > Tools > Server/Version Info

• look at the line that states "Server Up Time" (near top of page) and see what it shows for "load average"

The first set of numbers is most important and hopefully, it will always be under or close to 1.0 but even 3.0 can be acceptable in certain situations. If any higher and on a regular basis, you know the Server is overloaded which will cause all cause of problems.

Most current load numbers right now (11:49 CST) are 2.75, 1.92, 1.54
Session-Total:597-8621 (if that helps any)
I run an educational site, so our busier time for traffic is during the day.

I am now also receiving download errors (by redirect, streaming not working for some reason).

Two suggestions I can offer are:

• check your Hosting account Error logs to see if they provide any insight
• contact your Hoster to see if they can help with live troubleshooting of the problem

Webhost says I should mention to you all that my pdf downloads have upwards of 800 pages. I would think what matters is the overall size of the pdf which in this case is about 50-60MB for these files. I didn't have these consistent download problems with these same large files before. Direct links for these download fine. This morning I had trouble downloading a 224 pg 3MB file. It just stopped.

Does anyone know how the number of pages would adversely affect the download process? Also, this doesn't explain why my download by streaming is no longer working either (unless it has something to do with page headers RE: Dr. Byte's comment above).

Here's what my webhost says after they tested the files through the customer account area:

"This is what happened to me when I tried to download your files. When I clicked on the link, a new page opened up and my adobe pdf plugin in my browser tried to open the 800+ page in my browser. It choked about half-way through due to the amount of resources my own computer was using to try to open the file within the browser. When I went back to the download page and right-clicked on the download image and selected the save as file function, I was able to download the file with no problem.

As for server load, the server that you are on is a 2 cpu service which means the server load that you see should be divided by the number of cpus which in this case is 2. So if the server is running at a load of 3 then divide that number between the 2 cpus and the load per cpu is really around 1.5 which is not a problem. You typically would not see any adverse effects such as slow loading of a page unless the load was around 4-5. Even then the server would still be fully functional.

I monitored the server load while I was trying the open the pdf file within my browser and the load actually went down while I was trying to open the file in my browser. The server load was a little higher when I used the "save as file" function but I was able to download the file no problem that way.
With this testing, I am convinced the issue lies with the Adobe plugin and perhaps with the way you are creating your pdf files. Since I am able to open the pdf file in Adobe with no problem once I downloaded it, I think the issues lies with the Adobe plugin in the browser."

Even today, the same file (for me) would sometimes download w/o any errors and then the next time it would not.

How does that happen? Seems like it either should work or not work - not both.

Also, is it possible for direct links to work and then not for the redirect to work due to an Adobe plug-in issue? This is where I get lost in the technical processes of what's going on here.

Thanks.

Number of pages has no specific effect, other than the fact that the more pages you have, the larger the file that needs to be downloaded.

Have you tried a test where you use "Save As" instead of opening directly in the browser?

And, again, are your redirected downloads breaking when another customer starts another download while the first one is being downloaded?

And, for clarity, the "Adobe Plugin" is the adobe reader addon that is downloadable from the adobe website. Its only purpose in this equation is to load and display the file. Your browser is handling the communication to your webserver, not the plugin.

Debra_in_MO:

Webhost says I should mention to you all that my pdf downloads have upwards of 800 pages. I would think what matters is the overall size of the pdf which in this case is about 50-60MB for these files.
And I would say it is a Yes & No answer.

File size always has the higher importance but the format must also be taken into consideration. To me, it makes no sense having to use a PDF Reader to open and view a file I am going to download anyway. That is making people use two programs instead of one - just the Browser.

I'm sure Zen Cart documentation / information makes mention somewhere that all Download files should be in Zip format. If not then take note now. :smile:

By providing "all" downloads in Zip file format you ease the strain of resources used on the Server and make downloading much easier for your Customers.

I daresay if you start using Zip format for all your downloadable files, many of your download problems will disappear. :clap:

I am the web host in question here.

A simple search via Google reveals that there can be problems opening large PDFs (over 4mb) within the browser. I only mentioned the page numbers to indicate that these are very large pdfs and the adobe reader plugin still has to try to open all of these pages in the browser. At least in my test the document was in the process of being opened, i.e. I could go to the first several hundred pages while it was still being downloaded then it froze around the 450th page. The key thing to note however is that these are pdf files.

We had mentioned to our client in a previous support ticket they should look into a compression utility such as Winzip for their files.

I can say this: it is definitely not a resource issue with the server. I have tested her downloads and using her download script produces no noticeable load on the server, either cpu or memory. Also these problems are occurring on a server that on average runs between .75-1.25 server load which is low for a 2 cpu server. The only difference I have noted is that downloading the file chokes (no server spikes are noticeable) when the file is being opened within the browser via the adobe plugin but it downloads fine when using the "save file as" feature. It also downloads a lot faster when using the "save file as" instead of trying to open the document up in the browser.

I totally concur with Website Rob's that if the client uses the zip format for her files, then her download issues would cease.

Brian
EMWD.com

Website Rob:

I daresay if you start using Zip format for all your downloadable files, many of your download problems will disappear. :clap:

bcarpenter:

I totally concur with Website Rob's that if the client uses the zip format for her files, then her download issues would cease.

Using a zip format causes the browser to have to download the entire file to the PC before it is used. That tends to make the browser hold onto and monitor download communications more carefully until the file is fully downloaded. This in turn uses memory more efficiently, stops CPU spikes on the PC, and tends to prevent interruptions to the download connection. It also makes storing/archiving of the downloaded PDF (inside the zip) simpler. So, yes, generally speaking it's all 'round a good idea. Using "Save As" instead of just directly-clicking to download is also a good idea, but that's dependent on the customer knowing enough to do that.

So, generally speaking then, a 50MB zip will download more efficiently than a 50MB pdf. I was under the impression that the zipping was going to make my file size smaller and that was why the zip method is recommended. When I zip my files, they do not get more than a couple of MB smaller which doesn't matter much with a 50-60MB file. That's why I continued with pdfs. Sorry, I guess I had this all wrong.

The process of the "save as target" makes a lot of sense as compared to downloading the file in the browser. I'll add these directions to the site and begin testing it with our email list immediately. Can anyone confirm these directions for a Mac for the "save target as" method: "hold down the mouse button (Mac OS) on the link to the PDF file, and then choose Save Target As from the pop-up menu." Just curious if this is reliable info I should put on my site for Mac users.

**Also, I still have this problem:
If I change my attributes to download by streaming and turn off download by redirect, I immediately get an IE "can not display this page" error and Firefox just hangs up when trying to download a digital product. Dr. Byte said, "If it was truly in streaming mode at that point, that suggests that somehow the streaming isn't starting properly, whether a timeout or a lag in processing, or unexpected page headers are being sent, etc." How do I troubleshoot "unexpected page headers"? I'm still learning here and have no idea what that means. I'm not planning on using the streaming method, but I don't like that this is not working correctly as it may indicate I have something else incorrectly configured.

Thanks for all of your shared insight!

Debra_in_MO:

The process of the "save as target" makes a lot of sense as compared to downloading the file in the browser. I'll add these directions to the site and begin testing it with our email list immediately. Can anyone confirm these directions for a Mac for the "save target as" method: "hold down the mouse button (Mac OS) on the link to the PDF file, and then choose Save Target As from the pop-up menu." Just curious if this is reliable info I should put on my site for Mac users.In Safari, they choose "Download Linked File".
In Firefox, they choose "Save Linked File As"

Debra_in_MO:

**Also, I still have this problem:
If I change my attributes to download by streaming and turn off download by redirect, I immediately get an IE "can not display this page" error and Firefox just hangs up when trying to download a digital product. Dr. Byte said, "If it was truly in streaming mode at that point, that suggests that somehow the streaming isn't starting properly, whether a timeout or a lag in processing, or unexpected page headers are being sent, etc." How do I troubleshoot "unexpected page headers"? I'm still learning here and have no idea what that means. I'm not planning on using the streaming method, but I don't like that this is not working correctly as it may indicate I have something else incorrectly configured.
To find out what page-headers are sent when the page is downloaded, you need to find something that will log the stream of data sent from the browser as the file is downloaded. IE sucks at that. Firefox is okay with it when viewing pages, but is kinda tough to give access to that info when downloading files. Almost need a command-line browser like L ynx ...
That said, Zen Cart sends very conventional headers when it starts a download. If the wrong header is being sent, and you don't have any installed addons altering the download behavior, then I'd have to blame the server.

To check contents of the actual received file, you'd need a "hex editor" which allows you to view the file in binary format. And then you'd want to compare both the original file and the downloaded file to see if they start with the same content. If the downloaded file contains any HTML or lots of recognizable text, then something's amuck and is sending extra HTML content instead of only sending the expected file contents. Ultra Edit can view files in hex format, but it's not a free program. There are others out there that can do it too.