It seems to be an issue with the additional_images.php override found in:
includes/modules/YOUR_TEMPLATE_FOLDER
I fixed the problem by replacing the line (near line 23):
$products_image_base = str_replace($products_image_extension . '$', '', $products_image);
with:
$products_image_base = str_replace($products_image_extension, '', $products_image);
This fixes the problem and applies lightbox to all images, but produces a few javascript errors that were not there before:
showtrail is not defined
onmouseover(mouseover clientX=0, clientY=0) @ index.php (line 1)
hidetrail is not defined
onmouseout(mouseout clientX=0, clientY=0) @ index.php (line 1)
They don't seem to effect anything important on any browsers I tested on, but I do hate to have any javascript errors at all.