Questioning win rate confidence intervals

[Girchuck]

Lets say that you are a 3BB/100 winner over 25K hands.
If you estimate a 95% confidence interval of your true win rate, it would be approximately from 1 to 5 BB/100
However, in real life there are many more 1BB/100 winners than 5BB/100 winners. Intuitively, I think that 5BB/100 true win rate is less likely for a player with the above stats than 1BB/100 win rate. In other words, I doubt that the Gaussian distribution is unbounded on the right side.
Am I in error?

[RiverTheNuts]

This is accurate, it is not a normal distribution.. while it is not bounded, I'd imagine it certainly starts to taper off around 3 BB

Also, I didnt pay attention in probability last semester, but don't confidence intervals take the distribution curve into account??

[AaronBrown]

You're right, but the problem isn't the Normal distribution.

First of all, the 1 to 5 BB confidence interval implies you have a standard deviation of about 16 BB per 100 hands. That's unrealistically low, most player are 2 to 2.5 times that. But that doesn't matter for the principle here.

Suppose that poker playing talent is distributed among players according to a Normal distribution with an average of -1 BB/100 (rake) and a standard deviation of 2 BB. That means about two thirds of players are between -3 and +1 BB, 95% between -5 and +3, and only 1 out of 741 is at +5 BB or better. This is playing talent, which we assumes stays fixed.

1,000,000 of these players all play enough hands until the standard deviation of their performance is 1 BB/100 hands. I think that takes about 100K hands, but it doesn't matter.

3,612 of them will end up with +EV between 2.95 and 3.05 BB/100. Of those, only 3 will have true EV between 2.95 and 3.05. 4 of them will be +5 or better players who got unlucky. 563 will be between +3 and +5, 2,813 will be positive EV players but worse that +3 and 17 will actually be negative EV players who got lucky.

The 95% confidence interval for true EV is 0.6 to 3.6. The average EV of these players is +1.2.

I think this analysis is too extreme. I suspect that there are more +3 and even +5 EV players than the Normal distribution suggests. That's true of most things, there are more very tall and very short people than you would guess by looking at the standard deviation of height.

The classic statistics example of this effect is a drug company develops a test for a genetic condition that is 99% accurate. If you have the condition the test is positive 99% of the time, if you don't have the condition the test gives a false positive only 1% of the time. One person in a million has this condition.

Suppose you select a person at random and give them the test. It comes out positive. What is the probability that the person has the condition?

The answer is not 99%. Imagine we give the test to 100 million people. 100 of them have the condition, 99 of those test positive. 99,999,900 don't have the condition, 999,999 of them test positive. So only 99/1,000,098 of the people who test positive have the condition, about 0.01%. So even with the positive result from a 99% accurate test, there's a 99.99% chance you don't have the condition.