Eliminating dual content due to missing cPath variable

Good day,

Some time ago I read about search engines seeing links with and without cPath as different links, seeing this as dual content, this being a bad thing.

I don't know if that has been solved or not, but since I couldn't find anything about it and it was possible on my install I looked for a solution.

I adjusted a piece of code someone else on this board put together for me (originally so I could redirect old incoming quick-cart links to the new zen-cart links).

This code checks if a product_info link has a cPath, and if it doesn't it looks it up and redirects with a 301 moved permanently header.

So far it seems to work, if anyone spots any problems with this system I'd love to hear about it.

if ($_GET['main_page'] == 'product_info' && !(isset($_GET['cPath'])) && isset($_GET['products_id']) && (int)$_GET['products_id'] > 0) {
  $iProducts_id = (int)$_GET['products_id'];
  Header( "HTTP/1.1 301 Moved Permanently" );
  zen_redirect(zen_href_link(zen_get_info_page($iProducts_id), 'cPath=' . zen_get_product_path($iProducts_id).'&products_id='.$iProducts_id));
}

There's probably tons of places to put this and it'll work, I just put it in index.php below:

/**
 * Load common library stuff 
 */
  require('includes/application_top.php');"

Past the 7 minute editting time, just wanted to add that you should always backup and test rigorously before putting code on a live site.

Good luck!

Your code doesn't take into consideration the issue of what to do when a product is linked to more than one category.

Zen Cart v1.3.8 has the bulk, if not all, of the code using the same cpath and products_id ...
Have you tried this to see if it would work for you?

I read a thread not too long ago where it turned out that a box in which products that were bought along with a certain product by other people were shown did not include the cPath (in 1.3.8), didn't see a solution posted for that yet, but I haven't really searched.

I have tested this myself on 1.3.8 and it worked for me, didn't put it on our live site yet in case someone would reply with negative thoughts about the code, but I'll probably put it on tomorrow.

I've tried it specifically with a linked product (because I read about the problems with that in the thread about the centerbox mentioned above), and it used the master category as cPath, I'll try it again with some other linked products, but I expect it grabs the master category's cPath by default. (if you think this won't be the case, or there are other potential problems I'd love to hear it, naturally)

I can't think of any situations where our site would give a link without a cPath, but through Google Webmaster Tools I can see that there are some products indexed having a link with and without a cPath (they're shown as having duplicate meta tags \ titles), which is unwanted behavior, that's why I went for this.

It comes down to this being possible with the current incarnation of zen-cart (if this comes from a bug in a mod or someone placing a link and taking out the cPath doesn't matter, it's possible), while (I think), this code makes accessing such a link impossible and redirects it to a complete link. That can only be a good thing, right?

Diavire:

Good day,

Some time ago I read about search engines seeing links with and without cPath as different links, seeing this as dual content, this being a bad thing.

I don't know if that has been solved or not, but since I couldn't find anything about it and it was possible on my install I looked for a solution.

I adjusted a piece of code someone else on this board put together for me (originally so I could redirect old incoming quick-cart links to the new zen-cart links).

This code checks if a product_info link has a cPath, and if it doesn't it looks it up and redirects with a 301 moved permanently header.

So far it seems to work, if anyone spots any problems with this system I'd love to hear about it.

if ($_GET['main_page'] == 'product_info' && !(isset($_GET['cPath'])) && isset($_GET['products_id']) && (int)$_GET['products_id'] > 0) {
$iProducts_id = (int)$_GET['products_id'];
Header( "HTTP/1.1 301 Moved Permanently" );
zen_redirect(zen_href_link(zen_get_info_page($iProducts_id), 'cPath=' . zen_get_product_path($iProducts_id).'&products_id='.$iProducts_id));
}

>  
> There's probably tons of places to put this and it'll work, I just put it in index.php below:
> ```
/**
 * Load common library stuff 
 */
  require('includes/application_top.php');"

Diavire,
I have been looking for the solution to Google reported duplicate content for days... Thank you very much - your code seems to work perfectly.
I can only tell that it does convert the multiple product urls without the "cPath" to the one with the "cPath" for each product (for any product on my site, including products with multiple category links). It is too early (I just implemented your code earlier today) if Google is going to remove duplicate content warnings for my site.

I have one question however. Is there a way to actually see what search engine sees when it encounters one of the "HTTP/1.1 301 Moved Permanently" pages? I tried to look at the redirecting page code with a page snooper (utility from www.submitexpress.com) which is supposed to show you what Search Engine sees on your page but I couldn't find any header line with 301 redirect in it... How do I know search engine is really getting the 301 message? How does it work? Is the 301 redirect permanently command supposed to be "invisible"?

Here are my duplicate content links for one of my products

http://www.all4coffee.com/index.php?main_page=product_info&cPath=19&products_id=97

http://www.all4coffee.com/index.php?main_page=product_info&products_id=97

Ajeh:

Your code doesn't take into consideration the issue of what to do when a product is linked to more than one category.

Zen Cart v1.3.8 has the bulk, if not all, of the code using the same cpath and products_id ...
Have you tried this to see if it would work for you?

I've now put it live on my site, tested with some other linked products and it uses the main category for cPath in all cases, so far so good. I'd like to hear back from you wether or not using this is a good idea or not and for what reasons.

Maybe you also have some thoughts on what could be going wrong for all4coffee below?

all4coffee:

Here are my duplicate content links for one of my products

http://www.all4coffee.com/index.php?main_page=product_info&cPath=19&products_id=97

http://www.all4coffee.com/index.php?main_page=product_info&products_id=97

Glad you're happy with the code, but there seem to be some problems... somewhere...

First off, the first link seems to have an incomplete cPath? When following the second link it redirects to cPath 6_19, while the first link is just 19, did you mean to post it like that?

Basically, the cPath 19 is a problem, because since a cPath is given, the code doesn't think twice about it and doesn't check if the cPath is actually correct.
You could code around that and make it check the cPath even if one is given, but I don't know how heavy that's going to be on the server...

Secondly, when checking my headers using this code, I get a 301 Moved Permanently. When checking the redirect on your site, you get a 301 OK. That's not right. For a normal page you're supposed to get a "200 OK", for a permanent redirect that's "301 Moved Permanently", this is some weird combination of the two. I'd think that's something you need to contact your hoster about, because search engines aren't getting the right header, or even a valid one, when redirecting now.

BTW, forgot, you can check your headers by googling "header check", that'll give you a bunch of pages that'll show you what your site is reporting.

Diavire, here are the results of some testing I did, on the http://www.all4coffee.com/ site.

For the most part, everything below would apply to any Zen Cart site.

The URLs have been shortened for easier readability.

Sub-category Link
main_page=index&cPath=6_19

• Loose Tea Blends

Product Link
main_page=product_info&cPath=6_19&products_id=97

• European Blend Tea

All Listings
main_page=products_all&disp_order=1&page=5

• European Blend Tea is 7th on the page

Note: In this case the "All Listings" will always show a Product Page URL without a 'cPath', such as:
main_page=product_info&products_id=97

• don't think it is supposed to be that way though?

The following 'unknown' links all show "European Blend Tea"
main_page=product_info&cPath=6&products_id=97
main_page=product_info&cPath=19&products_id=97

As to how a SE will find the above 2 'unknown' links, I don't know.

IMHO, all the above is just more reasons why people should not worry about Duplicate Content.

Reference: Duplicate content

Duplicate content generally refers to substantive blocks of content within or across domains that either completely match other content or are appreciably similar. Mostly, this is not deceptive in origin. Examples of non-malicious duplicate content could include:

* Discussion forums that can generate both regular and stripped-down pages targeted at mobile devices
* Store items shown or linked via multiple distinct URLs
* Printer-only versions of web pages

And, as to the 301 OK msg., not a big deal. That's just how an MS Server operates. The number is what's important to Servers/Software and the following Text msg. is for us Humans. :wink:

Thanks for taking a closer look ;)

You're right in assuming the all products section shouldn't show links without cPaths, maybe he's using an old version of zen-cart? In any case, our site does show cPaths in that section.

The redirecting code does seem to be doing its work though, as all products I tried end up with a cPath.

It's also correct that both those links show European Tea Blend, as zen-cart only looks for the product id to show the product in question, you could set a cPath of eleventy trillion, and if that number matches a category it'll show the product to be in that category. You could make a plastic model kit show up in a wooden model kit category, just by changing the cpath, but that's of course not where you want it ;)

This also gives problems when you click the Category List button in a product, as you wind up in the wrong category.

How a SE would find those links is indeed a bit mysterious :P

It's probably true that Google doesn't care as much in this type of case, so likely no reason to worry, but if you can have your affairs in order, why not do so? :)

Thanks for the MS server bit, I didn't know that >)

Diavire:

I've now put it live on my site, tested with some other linked products and it uses the main category for cPath in all cases, so far so good. I'd like to hear back from you wether or not using this is a good idea or not and for what reasons.

Maybe you also have some thoughts on what could be going wrong for all4coffee below?

Glad you're happy with the code, but there seem to be some problems... somewhere...

First off, the first link seems to have an incomplete cPath? When following the second link it redirects to cPath 6_19, while the first link is just 19, did you mean to post it like that?

Basically, the cPath 19 is a problem, because since a cPath is given, the code doesn't think twice about it and doesn't check if the cPath is actually correct.
You could code around that and make it check the cPath even if one is given, but I don't know how heavy that's going to be on the server...

Secondly, when checking my headers using this code, I get a 301 Moved Permanently. When checking the redirect on your site, you get a 301 OK. That's not right. For a normal page you're supposed to get a "200 OK", for a permanent redirect that's "301 Moved Permanently", this is some weird combination of the two. I'd think that's something you need to contact your hoster about, because search engines aren't getting the right header, or even a valid one, when redirecting now.

Thanks for responding. The quoted links just pasted this way but when you look on display at the bottom of the browser they differ - anyway your code redirects perfectly the one without "cPath". Now when I know how to look up the server header info and that it returns the intended 301 (thanks Website Rob for explanation) I did some additional testing on my site and after checking all possible combinations (I also used the duplicate content links from Google webmaster tools - all seems to be working smoothly and seamlessly! Thank you so much for the code - I'm using 1.3.7 BTW and don't want to upgrade if I don't have to.

Maybe it is coincinental but until Google started pointing out duplicate tags, titles, content our site listed much higher. Thanks again everybody.

all4coffee:

Thanks for responding. The quoted links just pasted this way but when you look on display at the bottom of the browser they differ - anyway your code redirects perfectly the one without "cPath". Now when I know how to look up the server header info and that it returns the intended 301 (thanks Website Rob for explanation) I did some additional testing on my site and after checking all possible combinations (I also used the duplicate content links from Google webmaster tools - all seems to be working smoothly and seamlessly! Thank you so much for the code - I'm using 1.3.7 BTW and don't want to upgrade if I don't have to.

Maybe it is coincinental but until Google started pointing out duplicate tags, titles, content our site listed much higher. Thanks again everybody.

Great to hear it worked out well for you >)

One thing though, I wasn't talking about how the links looked, but the full links.
The first one you posted has "19" as cPath, and the second one (without cPath) redirected to "6_19", check it out.

Diavire:

Great to hear it worked out well for you >)

One thing though, I wasn't talking about how the links looked, but the full links.
The first one you posted has "19" as cPath, and the second one (without cPath) redirected to "6_19", check it out.

They lead to the same product... Is this a problem I should worry about?

all4coffee:

They lead to the same product... Is this a problem I should worry about?

I wrote about that in an earlier post in this thread, what happens is, nomatter what the cPath, as long as the product ID is the same, it'll show that product, the cPath decides what category is shown along with the product, so, by changing the cPath, the product is shown in a category that it's not supposed to be in.

I'll try and explain with example.

The numbers in the cPath variable tell the system what category to show.
"6" is "Coffee, Tea & Chocolate"
"19" is "Loose Tea Blends"
19 is a subcategory of 6, so "6_19" tells the system to show an opened "Coffee, Tea & Chocolate -> Loose Tea Blends"
Just "19" shows only "Loose Tea Blends", without "Coffee, Tea & Chocolate".

Functionally, this doesn't pose a problem, but the dual content issue isn't fixed, it's replaced with a different dual content issue.
First you had duplicates with a link with and without a cPath.
Now you have two different cPath links, one correct and one incomplete.

How the incomplete one got there I have no idea :P

I hope you understand what I'm trying to say, I had a bit of difficulty putting it into words ;)

Website Rob:

IMHO, all the above is just more reasons why people should not worry about Duplicate Content.

Reference: Duplicate content

Duplicate content generally refers to substantive blocks of content within or across domains that either completely match other content or are appreciably similar. Mostly, this is not deceptive in origin. Examples of non-malicious duplicate content could include:

* Discussion forums that can generate both regular and stripped-down pages targeted at mobile devices

* Store items shown or linked via multiple distinct URLs
* Printer-only versions of web pages

Note the highlighted comments above, from Google's own documentation:

Examples of non-malicious duplicate content could include:
* Store items shown or linked via multiple distinct URLs

I would add the following, taken from the same referenced document, which explains the reason for the hype:> However, in some cases, content is deliberately duplicated across domains in an attempt to manipulate search engine rankings or win more traffic. Deceptive practices like this can result in a poor user experience, when a visitor sees substantially the same content repeated within a set of search results.

...

Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results.

Since your store is not attempting to be deliberately deceptive in manipulating search engine results, there is less reason to be overly concerned about the matter. Google is not going to severely penalize you or bump you out of their indexed pages just because of having multiple links to the same content on your site.
That doesn't minimize the fact that it's prudent to keep duplication to a minimum. Google goes on to say that one of their reasons for desiring less duplicate content is so shoppers viewing search engine results don't have to sort through multiple listings of the same thing when looking at google pages, theoretically making their shopping experience more pleasant ... in Google's eyes anyway.

DrByte:

Note the highlighted comments above, from Google's own documentation:

Examples of non-malicious duplicate content could include:
* Store items shown or linked via multiple distinct URLs

I would add the following, taken from the same referenced document, which explains the reason for the hype:

Since your store is not attempting to be deliberately deceptive in manipulating search engine results, there is less reason to be overly concerned about the matter. Google is not going to severely penalize you or bump you out of their indexed pages just because of having multiple links to the same content on your site.
That doesn't minimize the fact that it's prudent to keep duplication to a minimum. Google goes on to say that one of their reasons for desiring less duplicate content is so shoppers viewing search engine results don't have to sort through multiple listings of the same thing when looking at google pages, theoretically making their shopping experience more pleasant ... in Google's eyes anyway.

I did read those, indeed, but as you said, always good to minimize potential problems, also, google isn't the only search engine, even if it's the by far most used one, and I don't know how other search engines deal with that kinda thing.

I do agree with the fact that, since none of this is deliberate, it's unlikely that it'll do bad things to your ratings ;D

Diavire:

I wrote about that in an earlier post in this thread, what happens is, nomatter what the cPath, as long as the product ID is the same, it'll show that product, the cPath decides what category is shown along with the product, so, by changing the cPath, the product is shown in a category that it's not supposed to be in.

I'll try and explain with example.

The numbers in the cPath variable tell the system what category to show.
"6" is "Coffee, Tea & Chocolate"
"19" is "Loose Tea Blends"
19 is a subcategory of 6, so "6_19" tells the system to show an opened "Coffee, Tea & Chocolate -> Loose Tea Blends"
Just "19" shows only "Loose Tea Blends", without "Coffee, Tea & Chocolate".

Functionally, this doesn't pose a problem, but the dual content issue isn't fixed, it's replaced with a different dual content issue.
First you had duplicates with a link with and without a cPath.
Now you have two different cPath links, one correct and one incomplete.

How the incomplete one got there I have no idea :P

I hope you understand what I'm trying to say, I had a bit of difficulty putting it into words ;)

I think I uderstand but this is really beyond my technical knowledge. Your code still seems to work and it did eliminate about 300 out of 1600 reported by Google "duplicate content" links. Today however I got this sitemap warning from Google:

URLs not followed
When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot because they contained too many redirects. Please change the URLs in your Sitemap that redirect and replace them with the destination URL (the redirect target). All valid URLs will still be submitted

And here are the "guilty" URL listed by Google (first URL in each pair is as it is listed by Google, the ones below in each pair I added as they are being displayed after being redirected by your code)

http://www.all4coffee.com/index.php?main_page=product_info&products_id=158

http://www.all4coffee.com/index.php?main_page=product_info&cPath=11&products_id=158

http://www.all4coffee.com/index.php?main_page=product_info&products_id=199

http://www.all4coffee.com/index.php?main_page=product_info&cPath=10_48&products_id=199

http://www.all4coffee.com/index.php?main_page=product_info&products_id=159

http://www.all4coffee.com/index.php?main_page=product_info&cPath=11&products_id=159

http://www.all4coffee.com/index.php?main_page=product_info&products_id=91

http://www.all4coffee.com/index.php?main_page=product_info&cPath=6_19&products_id=91

http://www.all4coffee.com/index.php?main_page=product_info&products_id=64

http://www.all4coffee.com/index.php?main_page=product_info&cPath=16&products_id=64

The above confuses me even more... I was under impression that the code redirects duplicates with 301 permanent redirect so Google takes a note of this and indexes only the one w/ cPath

Here is the link to my products Sitemap which Google took "samples" from:

http://www.all4coffee.com/sitemapproducts.xml

I'm using Google Sitemaps module (v.1.3.14) to generate my sitemaps. Should I upgrade to the latest vesion of Sitemaps (maybe generated links will look different on updated version of this mode? - I couldn't find any info on it). Also there is an Ultimate SEO mode - would this help? I'm lost.

all4coffee:

I think I uderstand but this is really beyond my technical knowledge. Your code still seems to work and it did eliminate about 300 out of 1600 reported by Google "duplicate content" links.

It may take some time for google to check all links, so more might be taken out with time. For me the number has been steadily declining over the past few days.

all4coffee:

Today however I got this sitemap warning from Google:

URLs not followed
When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot because they contained too many redirects. Please change the URLs in your Sitemap that redirect and replace them with the destination URL (the redirect target). All valid URLs will still be submitted

And here are the "guilty" URL listed by Google (first URL in each pair is as it is listed by Google, the ones below in each pair I added as they are being displayed after being redirected by your code)

http://www.all4coffee.com/index.php?main_page=product_info&products_id=158

http://www.all4coffee.com/index.php?main_page=product_info&cPath=11&products_id=158

---

http://www.all4coffee.com/index.php?main_page=product_info&products_id=199

http://www.all4coffee.com/index.php?main_page=product_info&cPath=10_48&products_id=199

http://www.all4coffee.com/index.php?main_page=product_info&products_id=159

http://www.all4coffee.com/index.php?main_page=product_info&cPath=11&products_id=159

http://www.all4coffee.com/index.php?main_page=product_info&products_id=91

http://www.all4coffee.com/index.php?main_page=product_info&cPath=6_19&products_id=91

http://www.all4coffee.com/index.php?main_page=product_info&products_id=64

http://www.all4coffee.com/index.php?main_page=product_info&cPath=16&products_id=64

The above confuses me even more... I was under impression that the code redirects duplicates with 301 permanent redirect so Google takes a note of this and indexes only the one w/ cPath

That is really weird..
I've compared your headers with the one from my own site, and they're no different, except for the 301 OK message (which is fine according to all search results I've found about it), and server specific info.

They're also the same messages as other redirects have given that I've been using for months on our site.

The only thing I've noticed is that these zen-cart redirects give two results on this header check site: http://www.seoconsultants.com/tools/headers.asp.
The first is the page that redirects and it's code, the second is the result of the page it was redirected to.

I made three small test php files that redirect to eachother (1 to 2, 2 to 3, 3 shows some text and it ends there), and those show two things.
1: When redirecting from the first testfile, it only shows info about the page it's redirected to, and no redirects after that. So if your site was redirecting multiple times, it should (I think), show only the first redirect. Since the header checker shows the correct redirected page for your site, it would seem that it doesn't redirect multiple times, as google seems to imply.

2: The test pages show only one result, where zen-cart redirects show two. The test page shows only the result from the page redirecting, and not the page it redirected to, while zen-cart links do.

To be honest I have no idea what any of that means, I'm just hoping someone more knowledgable can do something to help with that info.

all4coffee:

Here is the link to my products Sitemap which Google took "samples" from:

http://www.all4coffee.com/sitemapproducts.xml

I'm using Google Sitemaps module (v.1.3.14) to generate my sitemaps. Should I upgrade to the latest vesion of Sitemaps (maybe generated links will look different on updated version of this mode? - I couldn't find any info on it). Also there is an Ultimate SEO mode - would this help? I'm lost.

Dunno about Ultimate SEO, I've read nothing but trouble from those kinds of mods...

I don't see anything odd in the sitemap file either, linkwise, but upgrading might help, no idea though.

....hold on, just compared it to mine, and while the only real difference is that the graphical edges around the links look a little different... None of the links in my or your sitemap have any cPaths, what the hell? o_O

The above confuses me even more... I was under impression that the code redirects duplicates with 301 permanent redirect so Google takes a note of this and indexes only the one w/ cPath
That is a common fallacy amongst most people. Although each Search Engine has their own Rules they go by, for the most part, the URL one is Redirected to is not indexed. This is evidenced by the msg. Google provided, you.

When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot because they contained too many redirects. Please change the URLs in your Sitemap that redirect and replace them with the destination URL (the redirect target). All valid URLs will still be submitted

In other words; Replace the URLs that are being redirected with the 'real' URLs; the ones you end up on after being redirected.

I'm not sure why your 'xml' Site map has so many invalid URLs but you should replace them with actual/correct URLs; no redirection involved.

Website Rob:

In other words; Replace the URLs that are being redirected with the 'real' URLs; the ones you end up on after being redirected.

I'm not sure why your 'xml' Site map has so many invalid URLs but you should replace them with actual/correct URLs; no redirection involved.

I used the piece of code provided on the first page of this thread which changes/redirects URLs without "cPath" into ones with "cPath" and gives Google 301 redirect message... Sitemap was created with Google Sitemap automatically so I have no idea how would I replace "invalid URLs with actual/correct URLs" as you suggest... Should the URLs in the sitemap include the "cPath" bit or not? I'm really lost here trying to eliminate duplicate content on one side (which slowly seems to work with the aforementioned code provided in this thread), and on the other side fighting the Google too many redirects message...

I now submitted another sitemap generated with www.xml-sitemaps.com
utility, Google accepted it but hasn't indexed it yet so I don't know what their "comments" would be on this.
Here is the link to it:
http://www.all4coffee.com/sitemap.xml

Does this one look better in your opinion?