I don't see how that would work properly; except for the first group of products, that number will never be the per-page quantity. You could add a

[] Per Page

after the Displaying # to # (of ### products)

with the [] being the quantity dropdown.