I don't know How I can sort the products to be shown correctly when I click on Next and Previous

I have created a category with 50 products for example P1, P2,...,P50. I have set "Sort Order" for 4 products and set one number as "sort order" for Other products (i.e. [P20-SortOrder=31], [P21-SortOrder=32], [P22-SortOrder=33], [P23-SortOrder=34], and sort order for other products is 35). So when we go to this Category the sort of products is fine (i.e. P20-P21-P22-P23-P1-P2....P50) but when I click on one product and then click on next or previous the producst or not sorted (i.e. P20-P34-P41-....)

I don't know How I can sort the products to be shown correctly when I click on Next and Previous