If you leave off the doctype, you are put into quirks mode, in which browsers try to emulate bugs from the Netscape 4 and IE 5 era.
The biggest difference is probably the IE 5 box model (also called "traditional box model" or "border box"), which doesn't comply with the standard CSS box model. In the IE 5 box model, the size of boxes includes their borders and padding, while in the W3C box model, the size of boxes includes only the content of the boxes (the IE 5 version actually makes more sense, but it's not what's in the standard). In later versions of IE, the standard box model is used, but only if you're in standards mode; the IE 5 model is used in quirks mode. Nowadays, almost all browsers support a CSS3 property, box-sizing that can control which box model to use, so you can actually enable the traditional box model in a standard way (though it's prefixed -moz-box-sizing or -webkit-box-sizing in Gecko and Webkit, as that's the preferred way to implement standards that aren't yet finalized)
For more detail on the differences between quirks and standards mode, see: