Results 1 to 4 of 4
  1. #1
    Join Date
    Jan 2011
    Location
    Adelaide, Australia
    Posts
    1,668
    Plugin Contributions
    1

    Default Robots.text - Disallow: /cgi-bin/

    One of my newer sites has been published for 3 months without ranking at all in Google (does in Bing though)- robots.text had Disallow: /cgi-bin/ which contains sitemapxml.php - I do not have this folder disallowed on any of my other sites - GWT shows 353 pages indexed but just wondering if that disallow would have anything to do at all with the not ranking issue? I doubt it myself but am at a loss to ascertain why the site would not get a mention in search after such a length of time - same SEO etc applied as all my other sites, same template.

    site: https://www.cliponearrings.net.au

    cheers, (bewildered) Mike

  2. #2
    Join Date
    Apr 2006
    Location
    West Salem, IL
    Posts
    2,739
    Plugin Contributions
    0

    Default Re: Robots.text - Disallow: /cgi-bin/

    no, it has no bearing. Even if they were allowed in, there is nothing to see or index.
    Mike
    GeekHost - Zen Cart Certified & PCI Compliant Hosting
    The Zen Cart Forum...Better than a monitor covered with post-it notes!

  3. #3
    Join Date
    Dec 2011
    Location
    Rohnert Park, California, United States
    Posts
    8
    Plugin Contributions
    0

    Default Re: Robots.text - Disallow: /cgi-bin/

    Just did a search for "Fashion Clip On Earrings, Retro, Gold Plated with Grey Crystals" and your site came up first after the 3 sponsored ads.

  4. #4
    Join Date
    Jan 2007
    Location
    Australia
    Posts
    6,167
    Plugin Contributions
    7

    Default Re: Robots.text - Disallow: /cgi-bin/

    The cgi-bin folder should *always* be dissallowed. It should also be protected against bots (and people) that dont play by the rules via an .htaccess file.
    The file you mention should ideally be executed by a cron job on a daily basis. This php file will then create one or more xml files into a folder that is accessable to the bots. Typically there would be a "master" xml file in the root folder of the store that in turn will include/reference the other xml files (created by the php files in cgi-bin), that would typically reside in a zencart subfolder, called /sitemaps/. I use the word "typically" because it is user defined.

    FWIW, you don't *need* a site map for the zearch engines to crawl and index the site. They will do that with or without one. In the simplest terms, the site maps tell the SE's which pages you would like them to crawl 1st. They also tell the SE's when a page has changed (effectively asking them to recrawl/reindex). It doesnt force them to do anything, it is more of a polite request that they may or may not act upon.

    Cheers
    RodG

 

 

Similar Threads

  1. Replies: 5
    Last Post: 20 Jul 2011, 02:24 PM
  2. CGI-BIN folder needs to be created
    By [email protected] in forum Basic Configuration
    Replies: 6
    Last Post: 23 May 2006, 04:44 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
disjunctive-egg
Zen-Cart, Internet Selling Services, Klamath Falls, OR