My reason for thinking this is not psychological. Lets go to the extreme, if you suck and lose every hand you play in poker you will have a low variance (and a bad winrate). If you are winning all hands played you will also have a low variance. If you win half your hands and lose the other half, your variance will be high.
So being far away from the rest of the table in terms of skill will decrease your variance.
Is that reasoning correct? (I am not sure)