How confident can Little Johnny be in Confidence Intervals??

[Lexander]

Just adding a few thoughts.

A CI is not quite the same as a pure probability calculation. If all the underlying assumptions used to calculate the CI are correct, the actual estimated parameter will be within the interval 95% of the time if you are using an alpha of .05.

A CI will be wrong 5% of the time in this situation even if everything else you calculated is completely accurate. You have reason to suspect this sample is extreme. Your understanding of Poker sense tells you so. The CI calculation doesn't know any of this. It assumes the sample is just some random sample and makes no such assumptions.

Another way to think of it is to imagine every Poker player in the world performing this same calculation. What should happen is that about 95% of the calculated intervals should accurately estimate the actual rate of all those players. Little Johnny just happens to be one of those 5%.

As an aside, an alpha of 0.05 turns out to work pretty well in many cases. The lower the alpha, the less Type I errors you make but the increase in Type II errors means your Statistical Power decreases too quickly. Such is the tradeoff.

[JinX11]

[ QUOTE ]
If all the underlying assumptions used to calculate the CI are correct...

[/ QUOTE ]

Please elaborate, if you can.

I think confidence intervals assume a normal distribution. How are CIs effected by skewness (i.e., if the bell curve is larger towards one end of the curve)? On the other hand, is the affect that skewness may have on CIs countered by larger sample sizes?

If skewness does affect the validity of results using confidence intervals, is there any formulaic way to take this into account while producing similar results (that I am a% positive that the true value of x is between y and z)?

[shadow29]

The larger the sample is, the less the "skewed nature of the graph" would matter. Look up the Central Limit Theorem.

[JinX11]

Thanks, bro!

[Munga30]

Check out this thread for another treatment of your question.

[JinX11]

[ QUOTE ]
The larger the sample is, the less the "skewed nature of the graph" would matter. Look up the Central Limit Theorem.

[/ QUOTE ]

According to my reading, the Central Limit Theorem assumes normality and, in so doing, asserts that any skewedness experienced in a sample will go away in the long run. However, the more skewed the distribution (2 standard errors of skewedness seems to be referenced a lot), the less likely the distribution is truly normal; as such, less effective are the statistical tests that are based on normality (e.g., confidence intervals).

This appears to be the case in my actual scenario: the distribution I am testing is fairly heavily positively skewed. As such, the confidence intervals calculated are not very reliable and the daily fluctuations I see are to be expected.

[LinusKS]

[ QUOTE ]

We don't know whether Johnny is a winner or a loser. If he plays 7000 hands, dutifully does his confidence interval calculations, and then announces the 3BB - 10BB interval, we have to revise our initial estimate that he's most likely a loser. We do the sums (allow me to guess some answers here) and calculate that if he was a loser, there's a 2% chance of him registering that sort of winrate. If he's a winner, there's a 60% chance of that winrate range coming up.

So, consider 100,000 players. 99,000 of them are losers; of those guys, only 1980 will come up with such confidence intervals after their first 7000 hands. 1,000 players are winners and 600 of them will have such an interval. So, the chances that Johnny is actually a (temporarily) lucky loser are 1980 / ( 1980 + 600 ) or 77%.

Does that make you feel better?

[/ QUOTE ]

What he said.

(Minus the "feel better" part.)

Seriously, though, I'm the last thing from a math expert, but I think the Bayesian analysis is exactly right.

If you assume there's a large number of losers, and a small number of long-term winners, lucky losers will tend to outnumber true winners, even over a pretty good number of trials. Meaning confidence-interval analysis will tend to be misleading, at least when applied to a "random" player.

Yes?

[JinX11]

[ QUOTE ]
Check out this thread for another treatment of your question.

[/ QUOTE ]

Good link - I never hang out in Psych. Maybe I should start doing so....

[Lexander]

Jinx11,

Elaborating on some of your questions.

First, a CI does not assume a Normal Distribution. If I wanted a CI for the variance I would use a Chi-Squared Distribution much of the time.

The goal of a CI is to provide a reasonable range in which an unknown value should be found a certain percentage of time. The key here is that you don't know the actual Mean or Variance. It is often of interest to make inferences about what that unknown Mean is.

A CI based upon a Sample Mean (observed BB/hr) happens to be useful because it is a good estimator of the unknown Population Mean (actual BB/hr). The problem is that the observed value will almost never match the actual value.

However, when dealing with averages (such as observed BB/hr), the CLT provides an indication of how the different results will vary. More specifically, your observed BB/hr will behave like a Normal Distribution.

This is an important assumption. You don't actually get to see all the possible hands a person could ever play. You only get to see a small portion and you have to draw conclusions about the whole from that. Fortunately, because the CLT is true we can assume that a person's observed BB/hr can be modeled with a Normal Distribution.

We are also assuming that the conditions used to calculate the observed BB/hr are representative of the conditions you would expect in general. For example, if I calculate my observed BB/hr while playing with honorable friends it won't be much use in a game with a bunch of guys who stack the deck against me. In this case my estimate is probably going to be too high.

These are all assumptions, and if they turn out to be true, we can calculate a CI from those assumptions. It is called a Confidence Interval as opposed to a Probability Interval because of one key feature. When we calculate an observed BB/hr we are not dealing with a number. And a number is either in the interval or not.

Little Johnny has calculate a number. Assuming that all the other underlying assumptions for that CI are true, our calculation is useful. BTW, here are the assumptions my professors keep drilling into me:

1). We assume our samples have constant variance.
2). We assume our sampling distribution is distributed normally.
3.) We assume our samples are independent (or at least uncorrelated).

There are some concerns I have with these assumptions at a Poker table. For example, I have noticed that my samples are not independent. At a table of bad players I have better results than at stronger tables. If I am upset or sick I tend to play less efficiently.

I also think the constant variance assumption isn't true for a lot of people. Your variance in your samples depends on the nature of a given table and can be very different. Fortunately I don't think the variance is different enough to completely destroy the values, but it is a consideration.

The assumption of a normally distributed sampling distribution is pretty solid given the Central Limit Theorem. So it is really that independence assumption that bothers me in the CI calculations people do.

Anyhow, once you have your assumptions, you essentially have described your distribution. From that, you can construct a CI based on that distribution. If you want to know more about the construction of a CI, please let me know.

The final point to leave you on with a CI is that the 95% Confidence can be misleading. The interval either contains the actual value or it does not. Therefore you either get it right or wrong. With a 95% CI, you will on average get it wrong 5% of the time. This does not mean the CI procedure is wrong. The CI procedure is designed with the knowledge it won't get it right all the time.

[Alex/Mugaaz]

Can you recommend some intro books that could teach me:
A) What you're talking about?
B) How to figure out my own rates?

I'm pretty solid in the maths and sciences but I never studied anything like this.