"but still experienced page load times of 5-6s without caching"
But you do use caching right? As it can greatly increase to loading speed. For most online stores with the content not being changed all the time, caching can be really helpful.