Site's Search engine is very slow for site with 40,000 products

[catv]

My new zen cart has 40,000 products, and the site's search engine takes 30+ seconds to return a result. The results are usually accurate which I'm happy with but the speed of the search is a show stopper for us. Is there anyway to speed up the search? The old site's search was lighting fast with mysql full text searching. Doesn't zen cart take advantage of this? Are there any modules I can install related to site search? For speed improvments, site search statistics? I see the chemo made a search spider for OSC, is their a zen cart port of that?

I don't need really to search descriptions, how would I turn that off? I only need to search product name, model/ISBN, Author fields.

What have other people done to run big zen cart sites and handle search?

By the way if you have a big site, you'll have to turn off category counts, and hack the code for the products/categories admin listing so that it doesn't check for links since. I did this and went form 60 seconds to 1 second to open this tool in admin. Also need to turn side boxes and disable/hack admin tools to get rid of any drop down list of customers, products, manufactures. I mean come on image a drop down list with 40,000 items in it trying to load ahaha.

Anybody out there with a site with this many more products? What is the biggest store you have or have seen running zen cart?

[catv]

bump, we are suppose to the launch the site very soon, so we need to find a solution to speed up the site search urgently.

[catv]

hello, does no one want to reply?

[tj1]

What have other people done to run big zen cart sites and handle search?

Anybody out there with a site with this many more products? What is the biggest store you have or have seen running zen cart?

...well, I don't know if you'd consider my site big, it currently has about 20,000 products. Searches or just navigating the site used to take an awful long time. Solution, changed webhost; now searches take about 2 seconds or less.

I've heard some people here mentioning running ZC with 200,000 products.

[Ajeh]

I have a site with 700,000 products that I am looking at to speed up more ...

Already have pages from 30-45 seconds down to 0.3 seconds on most pages ... and 8 to 12 seconds on a couple pages that I am still looking at ...

Perhaps if you could post an URL to your site we can help with speeding this up ...

[catv]

My solution was to setup a full blown indexing site search engine, with nutch/lucene.

[Ajeh]

Do you have an URL that we could peek at your site to see what might have been the problem?

[catv]

Hi Ajeh, do you use the native zen search with the 700,000 product site??? That's crazy, I find that hard to believe. The zen search is just a bunch of plain sql queries, doesn't even use full-text indexing, I can't see how any site no matter how fast their server was have any success with the native search functionality when you have 20k+ products.

Why don't you change the queries to use full-text search, I think most people have graduated to mysql 3.x by now that supports it.

[Ajeh]

I did not say that was a bad idea ... just was curious to see the site and the solution you are using in action ...

[catv]

Hi ajeh, as soon as I get the staging site setup with the nutch search engine I'll let you take a peak, it's all in development/prototype right now, Hopefully tonight I'll run the first "clean" crawl of the site to create the index without all the noise urls in it.
Crawling takes time esp. with thousands of pages. Hopefully updating the index goes quicker. I think I can mark pages with timestamps in the metatags so that the nutch crawler skips the page if it hasn't changed.