When you do a default installation you also default to the Classic Template which uses tables for the sideboxes.

To use XHTML and CSS instead (highly recommended) you need to switch to a template, your own or somebody else's, that is based on the template_default.

To create your own template, all you would need to do is create a new directory in templates and then copy the template_info.php file from classic and make appropriate edits (they're pretty obvious when you open up the file) and also create a
css directory in the new template folder containing your stylesheet.css file. If you have changed any other tpl_ files e.g tpl_header.php, then you would also copy them from their directories under templates/classic to equivalent folders in your new template directory.

Once this is done, you go to Admin > Tools > Template Selection and invoke your new template. And then to Admin > Tools > Layout Boxes Controller to reselect the side boxes you had before, or press the reset button at the bottom to load up most of them in one go.

If that seems a bit overwhelming, an alternative is to use the Future Zen template which engages template_default for you (though you still need to reselect your side boxes) and gives you somewhere to start from. You can download it from the downloads section of this site and there's a link to the relevant downloads page in my signature below.