Getting upvoted is a positive experience. Getting downvoted is a negative one. If you don't know that you've been downvoted, you can't get upset about it. It's true that if you go back to a question, look for your relative position, and do the math, you may also have a negative experience, but that's much rarer.

Note that voting isn't the only thing than ranks answers. They also have some subtle reputation system going on.