How confident can Little Johnny be in Confidence Intervals??

[JinX11]

Cross-posted in Probability; no love, yet.

I should understand this; I don't. I'm interested for non-poker related reasons, but I can express my problem in poker terms. Suppose the following scenario; someone correct me when I go off track.

Little Johnny starts playing poker. Over his first 7,000 hands, he enjoys a terrific run of cards, winning at a 7BB/100 clip with a SD of 17 BB/100. Though Little Johnny rarely paid attention in class and routinely teased the teacher, he knows a little about statistics and thinks he knows how to calculate the range of his true win rate to a certain degree of confidence with Excel.

Using an alpha of 0.05 (95% confidence), he determines that with his sample size (7000 hands, or 7000/100 = 70 samples), present win rate (7 BB/100), and standard deviation 17 BB/100, little Johnny asserts that he is 95% confident that his maximum win rate is 10.98 BB/100 and his minimum win rate is 3.02 BB/100. Consequently, he quits his job and decides to play poker for a living.

We all know that his sample size is too small, but this scenario can easily occur. If anything, we are virtually 100% confident that his true win rate does NOT lie between 3.02 and 10.98 BB/100. Where did Little Johnny go wrong?


Thanks, in advance, for any responses. Any help is appreciated.

[That guy]

If anything, we are virtually 100% confident that his true win rate does NOT lie between 3.02 and 10.98 BB/100

not true... I assume you are trying to get at something regarding the inherent assumptions in statistics like std dev... (normality etc...)

[JoshuaD]

The statistics aren't wrong, they're just dealing with incomplete information.

Johnny knows that he's been on a good run, he's watched the cards and they've been falling better for him than they should.

So statistically, he's doin great 95% of the time, but Little Johnny knows that there's a good chance he's one of those 5% that aren't within that range.

[TStoneMBD]

there are many people who make 3BB/100 on these forums. im averaging over 3 at 3/6, but this is only with a 10k sample.

[JinX11]

Weird. I had a very similar run back in October of last year; I'm exactly 100% certain my true win rate is not between 3.02 and 10.98 BB/100.

On the other hand, I also had a month where my win rate was -1.82 BB/100 over 7000 hands, with a SD of about 13.5. According to these stats, my max true win rate is 1.34 BB/100 and my min true win rate is -4.98 BB/100. Notice that this range and the previous range do not overlap - they're not even close.

When we talk about the long run, we usually refer to many tens of thousands of hands. Certainly the range is very wide at 7k hands, but we can certainly acknowledge the values suggested by my initial example is absurd. Yet, unless I've calculated incorrectly, the mathematics support it; this is my problem.

[shadow29]

[ QUOTE ]
little Johnny asserts that he is 95% confident that his maximum win rate is 10.98 BB/100 and his minimum win rate is 3.02 BB/100.

[/ QUOTE ]

Wrong. Little Johnny's true win rate will fall (in repeated samples) in the calculated intervals.

[jtr]

I'm guessing you're reaching for some sort of Bayesian principle here, in which we assume that there are thousands of little Johnnies out there, and their true win rate is always something pretty awful, like zero, and that this particular little Johnny has failed to consider that possibility that there will always be a couple of lucky break-even Johnnies who run hot over their first 7000 hands and that he is one of these.

OK, fine. Some sort of Bayesian notion of what your win rate is likely to be in the absence of empirical information through playing is appropriate. That's why we laugh when someone says "My winrate is 87BB/100 over ten hands -- is this sustainable." 87 BB/100 is assumed to be ludicrously unlikely if not flat out impossible as a winrate.

Similarly, someone who read a lot of 2+2 postings, read some poker books, and knew some stats but who hadn't actually played might start out with an estimate of 1BB/100 for their winrate in a low-limit game. They'd be correct to only gradually adjust this estimate as more information from the real world came in (i.e., as they racked up thousands of hands).

But the 95% confidence interval that Johnny has correctly calculated is still true as far as it goes. It's just that the 5% contingency that his true rate is outside the confidence interval is not the same as zero, of course.

Also, your original unstated assumption that Johnny cannot possibly have a 3BB/100 or better winrate is debatable. Depends on the game, the opponents, how much he has studied / practiced, etc.

[JinX11]

[ QUOTE ]
there are many people who make 3BB/100 on these forums.

[/ QUOTE ]

Understood; I'll venture, however, noone is cranking 10+ BB/100, as the range max indicates is within the realm of possibility. I think(?) we can all acknowledge that 4BB/100 is probably the max possible.

If Little Johnny told you that he was 95% certain his win rate was between 3-10 BB/100, you'd laugh your head off, no? Also, if you make alpha = .01, this makes his min true win rate = 0.65BB/100. We all know that after only 7K hands, Little Johnny cannot say with 99% certainty that he is even a winning player.

At least that's the conventional wisdom........

[jtr]

Just saw your response -- no offence, but maybe the reason for your discomfort is that you're not totally clear on the "95%" part of the confidence interval.

The 95% interval and its complement, the 0.05 alpha-level in statistical testing, is based on a throwaway line by Ronald Fisher, the guy who put the basics of modern statistics together, that this might be a reasonable starting point for statistical hypothesis testing. In many areas of science but particularly in psychology, the 5% / 95% standard has come to be seen as something set in stone rather than the arbitrary cutoff point it really is.

If you want to calculate some confidence intervals vis a vis your own play that will restore your faith in the mathematics, try switching "95%" for "99.9%" (closer to most people's idea of "just about a dead certainty" anyway). Then you will find that the intervals are wide enough to keep you happy.

[JinX11]

[ QUOTE ]
Also, your original unstated assumption that Johnny cannot possibly have a 3BB/100 or better winrate is debatable. Depends on the game, the opponents, how much he has studied / practiced, etc.

[/ QUOTE ]

Actually, I fall in the boat that beieves 3BB/100+ is possible (esp at 2/4 Party). I failed to state my position correctly, I suppose.

I think we can probably agree that 4 or 5 or some higher value (call it x of BB/100 is simply NOT possible, though. However, it's relatively simple to have a short run where a 95% confidence interval examination suggests the min win rate that you can expect based on your short run is in excess of x.