If your store is configured with

Configuration->Product Listing->Display Product Listing Default Sort Order set to blank (product default sort order)
Configuration->Product Info->Previous Next - Sort Order set to 6 (product sort order)

and your store uses negative values for some of the product's sort-orders, then the order presented in the listing is different than the order presented within the "Previous/Next" display.

Specifically, if one product has a sort order of -1 and another has a sort order of -5, the sort in the listing is as expected (-5, -1, positive numbers) but the sort in the prev/next display isn't (-1, -5, positive numbers). The issue is that /includes/modules/product_prev_next.php uses:
Code:
    $prev_next_order= ' order by LPAD(p.products_sort_order,11,"0"), pd.products_name';
while /includes/index_filters/default_filter.php uses:
Code:
$listing_sql .= " order by p.products_sort_order, pd.products_name";
Changing the product_prev_next.php to remove the LPAD construct results in the same default sort order being presented.