How confident can Little Johnny be in Confidence Intervals??

JinX11 · #1 03-17-2005, 03:14 PM

Cross-posted in Probability; no love, yet.

I should understand this; I don't. I'm interested for non-poker related reasons, but I can express my problem in poker terms. Suppose the following scenario; someone correct me when I go off track.

Little Johnny starts playing poker. Over his first 7,000 hands, he enjoys a terrific run of cards, winning at a 7BB/100 clip with a SD of 17 BB/100. Though Little Johnny rarely paid attention in class and routinely teased the teacher, he knows a little about statistics and thinks he knows how to calculate the range of his true win rate to a certain degree of confidence with Excel.

Using an alpha of 0.05 (95% confidence), he determines that with his sample size (7000 hands, or 7000/100 = 70 samples), present win rate (7 BB/100), and standard deviation 17 BB/100, little Johnny asserts that he is 95% confident that his maximum win rate is 10.98 BB/100 and his minimum win rate is 3.02 BB/100. Consequently, he quits his job and decides to play poker for a living.

We all know that his sample size is too small, but this scenario can easily occur. If anything, we are virtually 100% confident that his true win rate does NOT lie between 3.02 and 10.98 BB/100. Where did Little Johnny go wrong?

Thanks, in advance, for any responses. Any help is appreciated.

That guy · #2 03-17-2005, 03:26 PM

If anything, we are virtually 100% confident that his true win rate does NOT lie between 3.02 and 10.98 BB/100

not true... I assume you are trying to get at something regarding the inherent assumptions in statistics like std dev... (normality etc...)

JinX11 · #3 03-17-2005, 03:39 PM

Weird. I had a very similar run back in October of last year; I'm exactly 100% certain my true win rate is not between 3.02 and 10.98 BB/100.

On the other hand, I also had a month where my win rate was -1.82 BB/100 over 7000 hands, with a SD of about 13.5. According to these stats, my max true win rate is 1.34 BB/100 and my min true win rate is -4.98 BB/100. Notice that this range and the previous range do not overlap - they're not even close.

When we talk about the long run, we usually refer to many tens of thousands of hands. Certainly the range is very wide at 7k hands, but we can certainly acknowledge the values suggested by my initial example is absurd. Yet, unless I've calculated incorrectly, the mathematics support it; this is my problem.

jtr · #4 03-17-2005, 03:48 PM

Just saw your response -- no offence, but maybe the reason for your discomfort is that you're not totally clear on the "95%" part of the confidence interval.

The 95% interval and its complement, the 0.05 alpha-level in statistical testing, is based on a throwaway line by Ronald Fisher, the guy who put the basics of modern statistics together, that this might be a reasonable starting point for statistical hypothesis testing. In many areas of science but particularly in psychology, the 5% / 95% standard has come to be seen as something set in stone rather than the arbitrary cutoff point it really is.

If you want to calculate some confidence intervals vis a vis your own play that will restore your faith in the mathematics, try switching "95%" for "99.9%" (closer to most people's idea of "just about a dead certainty" anyway). Then you will find that the intervals are wide enough to keep you happy.

JinX11 · #5 03-17-2005, 04:00 PM

Well, unless I am mistaken, in many legal arguments where regression analysis is conducted, an alpha of 0.05 is the legally accepted value.

As I originally mentioned, my purpose in asking is actually not poker related. I'm actually analyzing the performance (response times) of a website. This website sees about 100K hits per day with varying response times, depending on what page the user is hitting, what action the user is performing, etc. Each day, I calculate the average response time seen that day, along with a SD. The SD is about 3-4x the average response time, but because the sample size is very large, the 95% confidence interval is very small (somewhere around 1% of the average response time).

My problem is when I run this analysis on different days, the average response time and the min/max ranges don't intersect too often. This may not be the best application of regression analysis, because the usage on the system may differ from day to day (which I am ignoring, but I don't think the usage varies that much)....

[ QUOTE ]
you're not totally clear on the "95%" part of the confidence interval.

[/ QUOTE ]

Entirely possible. Once you perform a regression analysis using a 95% confidence interval, is it not correct to say, "I am 95% positive that the true x falls between y and z."?

EDITED: to add a missing word at the end of the first sentence.

jtr · #6 03-17-2005, 04:06 PM

[ QUOTE ]
Well, unless I am mistaken, in many legal arguments where regression analysis is conducted, an alpha of 0.05 is the legally accepted.

[/ QUOTE ]

I have little to do with the legal profession and have no idea whether this is true. But if it is, then the law is most certainly an ass.

(Interesting web-site data analysis problem deleted: in a nutshell I think your problem is too tight an interval brought about by too relaxed an alpha level -- you know you're always making tradeoffs between type I and type II errors, right?)

[ QUOTE ]
Entirely possible. Once you perform a regression analysis using a 95% confidence interval, is it not correct to say, "I am 95% positive that the true x falls between y and z."?

[/ QUOTE ]

Not quite but this is an extremely common and understandable misconception. What you can claim is that if the truth of the matter is that there is no relationship between x and the independent variables, then the chances of the observed variation (or more extreme variation) coming up simply due to chance are 5%. This is typically flipped around into the statement you gave, but they're not quite the same thing.

jtr · #7 03-17-2005, 04:12 PM

[ QUOTE ]
My problem is when I run this analysis on different days, the average response time and the min/max ranges don't intersect too often. This may not be the best application of regression analysis, because the usage on the system may differ from day to day (which I am ignoring, but I don't think the usage varies that much)....

[/ QUOTE ]

Bah. I really should leave this alone, sorry, but I have to respond to this bit. Your observation that the confidence intervals for different days are non-overlapping is pretty much equivalent to doing an analysis of variance of response time by day and finding a significant effect. So, how sure are you that there's no difference in response time across different days?

JoshuaD · #8 03-17-2005, 03:32 PM

The statistics aren't wrong, they're just dealing with incomplete information.

Johnny knows that he's been on a good run, he's watched the cards and they've been falling better for him than they should.

So statistically, he's doin great 95% of the time, but Little Johnny knows that there's a good chance he's one of those 5% that aren't within that range.

TStoneMBD · #9 03-17-2005, 03:39 PM

there are many people who make 3BB/100 on these forums. im averaging over 3 at 3/6, but this is only with a 10k sample.

JinX11 · #10 03-17-2005, 03:44 PM

[ QUOTE ]
there are many people who make 3BB/100 on these forums.

[/ QUOTE ]

Understood; I'll venture, however, noone is cranking 10+ BB/100, as the range max indicates is within the realm of possibility. I think(?) we can all acknowledge that 4BB/100 is probably the max possible.

If Little Johnny told you that he was 95% certain his win rate was between 3-10 BB/100, you'd laugh your head off, no? Also, if you make alpha = .01, this makes his min true win rate = 0.65BB/100. We all know that after only 7K hands, Little Johnny cannot say with 99% certainty that he is even a winning player.

At least that's the conventional wisdom........