[Xhad]

You're missing the fact that opponents who bluff you at the right frequency reduce your winrate and increase your variance no matter how you react to it. It's a lot harder to beat people when they're bluffing than when they're playing straightforwardly, to the point that David Sklansky states in Theory Of Poker that a table full of bad players who bluff correctly is tougher than a table full of mediocre players who play predictably.

Yes, their calling down makes you more likely to get paid off on good hands, but it also means you can't bluff, and more importantly their bluffing means you will have to give a lot of money back to them by paying them off when they do have a hand, lest you end up folding winners too often.