I guess you could attach a bit of javascript to the button to close the pop up page. I think that getting your shopping cart into the popup would be a bigger challenge. You'd be better off, putting it into a modal window and removing the continue shopping button in favor of the normal modal window close.
Either way you'd likely have some programming to do to get the shopping cart into an appropriate state to go into a pop up or a modal window, and with the popup you may have some session issues too if a user logged into the original window tried to checkout in the new one.
This can be done but would require a reasonably good knowledge of php, Zen Cart and javascript frameworks (I recommend jQuery).