Ban Google for a time

[michaelhl]

Hello,
because of the announcement from Dr. Byte I asked our provider to upgrade to PHP 5.2.4 and at the same time to MySQL 5..... First it seems, that the shop runs faster but the google BOT is permanently in the shop and because of the SEO-URLS it uses often the advanced-search_result.html. When this page ist called and also a page that doesen't exist, the shop stops for about ten minutes. I switched both search Boxes to off so there is no search but google uses that anymore. I add the link to the robots.txt but it has no effect. I try to delete Links from the webmaster-tools page at google, but they declined my wish to delete the missing pages, because the BOT gets no 404 Header. This is right, because the shop shows a page with no products when I click at google on a link to a not existing category.
Before we do the update of PHP and MySQL all works well but extremly slow. Now it is fast but often it isn't reachable because of the circumstances above.
So I want to ban google for a few days, that the BOT reads the robots.txt before it returns. Or shall I ask our provider for an downgrade of MySQL?

[DrByte]

- What version of Zen Cart are you using?
- In your /includes/languages/english/YOURTEMPLATE/meta_tags.php file, what do you have defined for ROBOTS_PAGES_TO_SKIP ?

[michaelhl]

This is the content of the ROBOTS_PAGES_TO_SKIP variable in metatags.php for the english and the german language:

login,logoff,create_account,account,account_edit,account_history,account_history _info,account_newsletters,account_notifications,account_password,address_book,ad vanced_search,advanced_search_result,checkout_success,checkout_process,checkout_ shipping,checkout_payment,checkout_confirmation,cookie_usage,create_account_succ ess,contact_us,download,download_timeout,customers_authorization,down_for_mainte nance,password_forgotten,time_out,unsubscribe

The used shop-version is 1.3.7.1. but it is massive changed by contributions. It contains the RSS-Feeds, the google-sitemap, tabbed products, multilg. EZ-Pages, Better Categories, SEO (update by Dr. BYte) and many more. But the fundamental change is the book contribution by harreslade because this gives a lot of customizeable fields for the products. The Book-contribution is changed for selling DVDs.
Our Provider sends me a slow-query.log file and when I have a look to it, it shows beside others of the same kind this query:

select count(distinct p.products_id) as total FROM
products p LEFT JOIN manufacturers m USING ( manufacturers_id ) ,
products_description pd
WHERE
p.products_status = 1
AND p.products_id = pd.products_id AND pd.language_id = 1
AND (
p.products_id IN (
SELECT DISTINCT
p.products_id
FROM (products p 
LEFT JOIN manufacturers m
USING(manufacturers_id), products_description pd, categories c, products_to_categories p2c )
LEFT JOIN meta_tags_products_description mtpd
ON mtpd.products_id= p2c.products_id
AND mtpd.language_id = 1
WHERE (p.products_status = 1
AND p.products_id = pd.products_id
AND pd.language_id = 1
AND p.products_id = p2c.products_id
AND p2c.categories_id = c.categories_id AND ((pd.products_name LIKE '%original Beschreibung%'
OR p.products_model
LIKE '%original Beschreibung%'
OR m.manufacturers_name
LIKE '%original Beschreibung%' OR (mtpd.metatags_keywords
LIKE '%original Beschreibung%'
AND mtpd.metatags_keywords !='') OR (mtpd.metatags_description
LIKE '%original Beschreibung%'
AND mtpd.metatags_description !='') OR pd.products_description
LIKE '%original Beschreibung%') ))
)
OR p.products_id IN (
SELECT DISTINCT
p.products_id
FROM
products p LEFT JOIN manufacturers m USING ( manufacturers_id ),
products_description pd,
categories c,
products_to_categories p2c LEFT JOIN meta_tags_products_description mtpd ON mtpd.products_id = p2c.products_id AND mtpd.language_id = 1,
product_book_extra pbe,
books_to_authors bta,
book_authors ba,
book_authors_info bai
WHERE
p.products_status = 1
AND p.products_id = pd.products_id
AND pd.language_id = 1
AND bta.language_idta = pd.language_id
AND ba.language_ida = pd.language_id 
AND p.products_id = p2c.products_id
AND p2c.categories_id = c.categories_id
AND pbe.products_id = p.products_id
AND bta.products_id = p.products_id
AND bta.book_authors_id = ba.book_authors_id
AND ba.book_authors_id = bai.book_authors_id
AND bai.language_id = pd.language_id
AND ((pd.products_name LIKE '%original Beschreibung%'
OR p.products_model LIKE '%original Beschreibung%'
OR m.manufacturers_name LIKE '%original Beschreibung%'
OR pd.products_description LIKE '%original Beschreibung%'OR ba.book_authors_name LIKE '%original Beschreibung%') )
)
OR p.products_id IN (
SELECT DISTINCT
p.products_id
FROM
products p LEFT JOIN manufacturers m USING ( manufacturers_id ),
products_description pd,
categories c,
products_to_categories p2c LEFT JOIN meta_tags_products_description mtpd ON mtpd.products_id = p2c.products_id AND mtpd.language_id = 1,
books_to_genres btg,
book_genre bg,
book_genre_description bgd
WHERE
p.products_status = 1
AND p.products_id = pd.products_id
AND pd.language_id = 1
AND btg.language_idg = 1 
AND p.products_id = p2c.products_id
AND p2c.categories_id = c.categories_id
AND btg.products_id = p.products_id
AND btg.book_genre_id = bg.book_genre_id
AND bg.book_genre_id = bgd.book_genre_id
AND bgd.language_id = 1
AND ((pd.products_name LIKE '%original Beschreibung%'
OR p.products_model LIKE '%original Beschreibung%'
OR m.manufacturers_name LIKE '%original Beschreibung%'
OR pd.products_description LIKE '%original Beschreibung%'
OR bgd.book_genre_name LIKE '%original Beschreibung%') )
)
OR p.products_id IN (
SELECT DISTINCT
p.products_id
FROM
products p LEFT JOIN manufacturers m USING ( manufacturers_id ),
products_description pd,
categories c,
products_to_categories p2c LEFT JOIN meta_tags_products_description mtpd ON mtpd.products_id = p2c.products_id AND mtpd.language_id = 1,
books_to_types btt,
book_type bt,
book_type_description btd
WHERE
p.products_status = 1
AND p.products_id = pd.products_id
AND pd.language_id = 1
AND btt.language_idt = 1 
AND p.products_id = p2c.products_id
AND p2c.categories_id = c.categories_id
AND btt.products_id = p.products_id
AND btt.book_type_id = bt.book_type_id
AND bt.book_type_id = btd.book_type_id
AND btd.language_id = 1
AND ((pd.products_name LIKE '%original Beschreibung%'
OR p.products_model LIKE '%original Beschreibung%'
OR m.manufacturers_name LIKE '%original Beschreibung%'
OR pd.products_description LIKE '%original Beschreibung%'
OR btd.book_type_name LIKE '%original Beschreibung%') )
)
);

Maybe that this have no effect for PHP 4.7 and MySQL 4 but after the Upgrade this kind of query stops the Database.
But why take the metatag.php and the robots.txt no effect? What can I do to forbit this kind of queries? May it take effect if I delete the class.productBookSearch.php from the classes/observers directory?

[DrByte]

What is your site URL?
Do you have links to predefined searches somewhere in your site navigation?

[michaelhl]

Thanks for your help. I solve the problem by deleting the search configure from the autostart folder -- was no zen-cart problem, it was from the book contribution. Google re-reads the robots.txt and ignore now the advanced_search_results. All works well now.