How confident can Little Johnny be in Confidence Intervals??

[shadow29]

I firmly believe that a 4-6 BB/100 is certainly possible for Party .5/1.

Also, nanolimit players might be able to boast a 7-10 bb/100 that is statistically significant.

[jtr]

[ QUOTE ]
If Little Johnny told you that he was 95% certain his win rate was between 3-10 BB/100, you'd laugh your head off, no?

[/ QUOTE ]

OK, this gets to the crux of it. If I had zero background knowledge about poker and winrates, I certainly wouldn't laugh. I'd say that Johnny was making a perfectly legitimate estimate under the circumstances. The reason you want to laugh at Johnny is that your have a much richer a priori idea than he does about what win rates are likely and/or possible. If we factor that into the calculations in a Bayesian way then we formally capture your intuition here.

Here's a simple example: let's say there are two kinds of poker players in the universe. There are long-run minor losers (99%) and big winners (1%). Winners make 4BB/100; losers lose enough to pay the winners plus the rake. Something like -0.5 BB/100 I guess.

We don't know whether Johnny is a winner or a loser. If he plays 7000 hands, dutifully does his confidence interval calculations, and then announces the 3BB - 10BB interval, we have to revise our initial estimate that he's most likely a loser. We do the sums (allow me to guess some answers here) and calculate that if he was a loser, there's a 2% chance of him registering that sort of winrate. If he's a winner, there's a 60% chance of that winrate range coming up.

So, consider 100,000 players. 99,000 of them are losers; of those guys, only 1980 will come up with such confidence intervals after their first 7000 hands. 1,000 players are winners and 600 of them will have such an interval. So, the chances that Johnny is actually a (temporarily) lucky loser are 1980 / ( 1980 + 600 ) or 77%.

Does that make you feel better?

[JinX11]

Well, unless I am mistaken, in many legal arguments where regression analysis is conducted, an alpha of 0.05 is the legally accepted value.

As I originally mentioned, my purpose in asking is actually not poker related. I'm actually analyzing the performance (response times) of a website. This website sees about 100K hits per day with varying response times, depending on what page the user is hitting, what action the user is performing, etc. Each day, I calculate the average response time seen that day, along with a SD. The SD is about 3-4x the average response time, but because the sample size is very large, the 95% confidence interval is very small (somewhere around 1% of the average response time).

My problem is when I run this analysis on different days, the average response time and the min/max ranges don't intersect too often. This may not be the best application of regression analysis, because the usage on the system may differ from day to day (which I am ignoring, but I don't think the usage varies that much)....

[ QUOTE ]
you're not totally clear on the "95%" part of the confidence interval.

[/ QUOTE ]

Entirely possible. Once you perform a regression analysis using a 95% confidence interval, is it not correct to say, "I am 95% positive that the true x falls between y and z."?

EDITED: to add a missing word at the end of the first sentence.

[JinX11]

[ QUOTE ]
I firmly believe that a 4-6 BB/100 is certainly possible for Party .5/1.

Also, nanolimit players might be able to boast a 7-10 bb/100 that is statistically significant.

[/ QUOTE ]

I'm not sure the limit Little Johnny was playing at when he had his great run has an influence on the mathematics.

[jtr]

[ QUOTE ]
Well, unless I am mistaken, in many legal arguments where regression analysis is conducted, an alpha of 0.05 is the legally accepted.

[/ QUOTE ]

I have little to do with the legal profession and have no idea whether this is true. But if it is, then the law is most certainly an ass.


(Interesting web-site data analysis problem deleted: in a nutshell I think your problem is too tight an interval brought about by too relaxed an alpha level -- you know you're always making tradeoffs between type I and type II errors, right?)


[ QUOTE ]
Entirely possible. Once you perform a regression analysis using a 95% confidence interval, is it not correct to say, "I am 95% positive that the true x falls between y and z."?

[/ QUOTE ]

Not quite but this is an extremely common and understandable misconception. What you can claim is that if the truth of the matter is that there is no relationship between x and the independent variables, then the chances of the observed variation (or more extreme variation) coming up simply due to chance are 5%. This is typically flipped around into the statement you gave, but they're not quite the same thing.

[jtr]

[ QUOTE ]
My problem is when I run this analysis on different days, the average response time and the min/max ranges don't intersect too often. This may not be the best application of regression analysis, because the usage on the system may differ from day to day (which I am ignoring, but I don't think the usage varies that much)....


[/ QUOTE ]

Bah. I really should leave this alone, sorry, but I have to respond to this bit. Your observation that the confidence intervals for different days are non-overlapping is pretty much equivalent to doing an analysis of variance of response time by day and finding a significant effect. So, how sure are you that there's no difference in response time across different days?

[JinX11]

[ QUOTE ]


(Interesting web-site data analysis problem deleted: in a nutshell I think your problem is too tight an interval brought about by too relaxed an alpha level -- you know you're always making tradeoffs between type I and type II errors, right?)

[/ QUOTE ]

First, I have no idea what Type I and II errors are. I need to research this.

Even when I make the alpha very small, the confidence interval stays very small in relation to the experienced average (unless Excel is not calculating something correctly).

In case you are curious:

SD: 5000
Sample: 100000
Alpha: 0.05

When I run CONFIDENCE(Alpha, SD, Sample), I get about 30. When I change alpha to 0.00001 (99.999% confidence), the CI only goes up to 70. The experienced average is about 1700, so the CI remains very small in relation to the average, implying(?) the experienced average is very close to the true value.

[JinX11]

[ QUOTE ]

So, how sure are you that there's no difference in response time across different days?

[/ QUOTE ]

No, that's just it: the average response time seen on one day rarely falls within the min/max range calculated via a 95%-CI from the previous day. I'm trying to understand how this is happening; ideally, I'd like to be able to report that I am x% certain that the performance the system is seeing is between y and z seconds. Maybe this isn't quite what is meant by CIs.

Anyhow, that's when I remembered my poker experience. The two seemed similar: I've had excellent short-term runs in poker that a 95%-CI suggests that my minimum true win rate exceeds what I know it to be; I've had bad short-term runs that a 95%-CI suggests that my maximum true win rate is lower than what I know it to be.

Just trying to understand these two different experiences can coexist mathematically.

[jtr]

On the subject of Type I and Type II errors, take a look at this page. Basically a Type I error is to believe that your wife is having an affair when she isn't really, and a Type II error is to believe that nothing's going on when in fact your wife is having an affair.

The alpha level that you set is the probability of experiencing a Type I error, i.e., incorrectly accepting some experimental hypothesis. You can set a tougher alpha level (0.01 instead of 0.05 for example) and thus avoid Type I errors with greater confidence, but in doing so you increase the probability of getting a Type II error (missing the truth of an experimental hypothesis).

[ QUOTE ]

No, that's just it: the average response time seen on one day rarely falls within the min/max range calculated via a 95%-CI from the previous day.

[/ QUOTE ]

OK, well if it looks like a duck, etc. I'd say you have some excellent evidence here for daily variation in response times. Where that variation is coming from is another matter, and you might want to pursue further regression analyses to figure out whether the daily variation correlates with anything else or whether there's just a mysterious surge of activity on Tuesdays, for example.


[ QUOTE ]
I'm trying to understand how this is happening; ideally, I'd like to be able to report that I am x% certain that the performance the system is seeing is between y and z seconds. Maybe this isn't quite what is meant by CIs.

[/ QUOTE ]

No, you seem to be approximately OK in your conception of what a CI is. But why are you comparing across days anyway if you feel there's no reason to expect a difference? If the Tuesday data and the Wednesday data differ in no sensible way (or if I'm uninterested in possible differences between them) I would aggregate them.

Unlike most poker players, you have a refreshingly huge sample size here, and so your confidence intervals are becoming very narrow, even when you choose a more rigorous standard than 95%. OK, that's to be expected.

If you can say that Tuesday's mean response time was 1705 with a 99% CI of plus/minus 15, then we really are quite confident that Tuesday's data is drawn from a distribution with a mean very close to 1705. Great. If Wednesday's data has a different mean, so be it. The universe is trying to tell you something -- I am not sure why you are resisting the message. (Not trying to be rude here, just not sure I have grasped the reason for your suspicion of the daily variation.)

[JinX11]

[ QUOTE ]

No, you seem to be approximately OK in your conception of what a CI is. But why are you comparing across days anyway if you feel there's no reason to expect a difference? If the Tuesday data and the Wednesday data differ in no sensible way (or if I'm uninterested in possible differences between them) I would aggregate them.

Unlike most poker players, you have a refreshingly huge sample size here, and so your confidence intervals are becoming very narrow, even when you choose a more rigorous standard than 95%. OK, that's to be expected.

If you can say that Tuesday's mean response time was 1705 with a 99% CI of plus/minus 15, then we really are quite confident that Tuesday's data is drawn from a distribution with a mean very close to 1705. Great. If Wednesday's data has a different mean, so be it. The universe is trying to tell you something -- I am not sure why you are resisting the message. (Not trying to be rude here, just not sure I have grasped the reason for your suspicion of the daily variation.)

[/ QUOTE ]

Very well rationalized and put; not rude at all. Let me think through all of this and read your link.



Again, thanks all!