Not really all that weird. PHP manual says:
The closing quote and semi-colon print because those are not code in php mode.
The "one-line" comment styles only comment to the end of the line or the current block of PHP code, whichever comes first. This means that HTML code after // ... ?> or # ... ?> WILL be printed: ?> breaks out of PHP mode and returns to HTML mode, and // or # cannot influence that. If the asp_tags configuration directive is enabled, it behaves the same with // %> and # %>. However, the </script> tag doesn't break out of PHP mode in a one-line comment.
So, I have to hold my hand up and say my previous comment was inaccurate. But when I run your code I see exactly what I would expect output to the page. And I'd add that it is pretty common practice to comment either on separate lines or after the end of a line of code.
The source code I get when I run that is:
Which seems completely right. I exclude the php info section as it just introduces two <html> tags to the document
<title>PHP test page</title>
<h1>Welcome to the PHP test page</h1>echo "<p>What is on this line?</p>";
<p>Back in PHP?</p><h2>Another PHP test</h2><p>Because we switch back to PHP immediately after the missed end tag, we are okay here.</p></body>