It's the same reason why when you're facing someone you don't agree on which way right (or left) is, but you do agree on up or down. The mirror doesn't flip anything, so when you look at it you see your right side on the right, but someone looking at you from the front would see your right side on the left.