If we follow your logic, we shouldn't count any card that has no chance of being dealt to us.

You're thinking that it's impossible to dealt a card that one of your opponent has. That's true, but it's just as impossible to dealt a card that's below the river. (cheating not withstanding)