Nothing in your shop accounts for 6 seconds ...

NOTE: a site that has not been touched ... as in no one has visited it for XXX amount of time, this is server dependant, does have MySQL queries that are not "stored" or preloaded by other users and there could be a delay ...

What this means is ...

Let's say I come to see your site today and am 1st in the door ...

I hit the All Products link ... and say this takes 6 seconds ...

I then click Page 2 of All Products ... and let's say this is 6 seconds ...

I then click Page 3 of All Products ... and let's say this is 6 seconds ...

Now I click back to page 2 ... and it takes 0.2 seconds ...

This is cached queries are not helping make the site fast ... some sites use this ... some do not ... others have it set very low ...

Now if each page regardless of going:

All Products Link
Page 2
Page 3
Page 2
Page 3

are always the same slow time ... then you have a dog of a server as your hosting site is not even attempting to optimize things for MySQL ... or the settings are too low and you need to talk to your hosting site and get them raised ...

Active sites have the advantage of more things are cached in the MySQL to keep things running fast ...

I am not a server type ... just talk to my hosting guru to understand how it is his servers are so fast and become faster with each click ... and the next guy in gets the fast queries that I made by looking around the site ...

Hope this might help too ...