Having position: could it be mutually beneficial to swap places?

[RoundTower]

If he has the choice between two empty seats, one to my left and one to my right, he should choose to sit on my left. This is accepted poker wisdom and I believe it is true here -- he will be in proportionally a lot of pots with me. Similarly it is thought that I should prefer the seat on his right.

So my question is - is this really more profitable for both of us? And if so how does having us in this configuration hurt the other players at the table? Is it really possible they will lose more money when I sit on the rock's right than vice versa?