Confidence Intervals and evaluating win rate

[magic_man]

We frequently see posts on the low-limit forum and other places in which the poster says they have calculated a win rate of 5 BB/hr over 100 hours (or whatever), and everyone laughs and says that 100 hours is not nearly long enough to calculate valid statistics. I am confused as to why a confidence interval cannot be used to validate these numbers. For example, suppose you play 6 sessions of 10-20 with the following results:

Net # Hours
-200 6
+600 6
+300 6
+300 6
-200 6
+500 6

Your win rate here would be:
$36/hr or just over 1.5 BB/hr

Your std deviation would be:
$128/hr or just over 6 BB/hr

This is all over 36 hours of play. If we calculate a 90% confidence interval for win rate, we get:

error = 1.64 * $128 / sqrt(36)
error = $35/hr

Therefore win rate is between $1 and $71 / hr.

So we can say with at least 90% confidence that this player is at least break-even. Is there an error in this argument? It seems to me that there would be many cases where after only one or two hundred hours, you could make a 95 or 99% confidence interval to show that you were a winning player. Am I making some mistake in my understanding of the equations used here? Thanks for any input you may have.

~Magic_Man

[BruceZ]

We frequently see posts on the low-limit forum and other places in which the poster says they have calculated a win rate of 5 BB/hr over 100 hours (or whatever), and everyone laughs and says that 100 hours is not nearly long enough to calculate valid statistics.

You can make confidence intervals after a small number of hours, as long as you keep in mind that they only apply to the conditions that were in effect when you played. It is likely that 100 hours may not be a large enough sample to adaquately represent the future if conditions are likely to change, for example, if you have a different mix of opponents in the future who play significantly differently than the opponents over the 100 hours. Still you can construct confidence intervals for the conditions which you did play against during those 100 hours.


So we can say with at least 90% confidence that this player is at least break-even. Is there an error in this argument?

Firstly, if this were the 90% confidence interval, then there would be a 95% confidence that the player is at least break-even since for this we want the 1-sided confidence interval, and only 5% falls below 0. Secondly, your calculations would be perfect if your estimate of the standard deviation were perfect. The SD converges fast enough that this is usually a minor factor, but for only 6 sessions the uncertainty in the SD becomes significant. To correct for this, you need to use the t-distribution. First multiply your SD by a factor of sqrt(6/5). Then for a 90% confidence interval, you would use the inverse t-distribution with 5 degrees of freedom. In Excel this is TINV(0.1,5) = 2.015. Use this instead of 1.64. These two corrections will cause your actual 90% confidence interval to widen by 35%, to +/- $47, instead of +/- $35. So it would go from -$11 to $83. 0 would lie at the edge of the 80% confidence interval, so there would be an 85% confidence of being a winning player.

[BruceZ]

0 would lie at the edge of the 80% confidence interval, so there would be an [/color:red]85%</font> confidence of being a winning player.

That should be 90% since 10% is below 0. You had said 90% originally, was that due to self-cancelling errors, or did you anticipate the effect of the uncertainty in SD?

[magic_man]

[ QUOTE ]

That should be 90% since 10% is below 0. You had said 90% originally, was that due to self-cancelling errors, or did you anticipate the effect of the uncertainty in SD?

[/ QUOTE ]

That was just my mistake originally.

I'd forgotten I should use the t-test...I don't remember this sqrt(6/5) thing though, where does that come from? I thought that the t-test essentially just gave a different value to use for Z.

~Magic_Man

[BruceZ]

I'd forgotten I should use the t-test...I don't remember this sqrt(6/5) thing though, where does that come from?

That's because you used the maximum likelihood estimator for the variance, which divides by N, so you have to multiply the estimate of sigma by the factor sqrt(N/N-1) before you can use the t-distribution. Alternatively, you could have used the unbiased estimator for the variance, which divides by N-1, and then you can use the square root of this directly. If sigma were known, we would use the normal distribution, and the fact that
sqrt(N)*(Xn-u)/sigma is a normally distributed random variable, where u is the mean, and Xn are the samples. Since sigma is unknown, we are replacing sigma with an estimate of sigma, est(sigma), and then sqrt(N)*(Xn-u)/est(sigma) is distributed as a t-distribution with N-1 degrees of freedom instead of a normal distribution, but it is only a t-distribution if we use the variance estimate that divides by N-1, or equivalently, by the maximum likelihood estimate that divides by N, with sigma scaled by sqrt(N/N-1). This becomes a negligible factor as N becomes reasonably large.

[BruceZ]

sqrt(N)*(Xn-u)/sigma is a normally distributed random variable, where u is the mean, and Xn are the samples.

Xn is the sample average. There's supposed to be a bar over it. &lt;Xn&gt; for physicists.

[magic_man]

Thanks for all the help...college stat class is all coming back to me now. All this being said, if I want to add a confidence interval calculation to my records, should I just wait until I have a "large" number and use a standard confidence interval, or should I use a t distribution always? As I understand it, once I reach a large number these intervals will be the same anyway, correct?

Thanks,
Magic_Man

[BruceZ]

All this being said, if I want to add a confidence interval calculation to my records, should I just wait until I have a "large" number and use a standard confidence interval, or should I use a t distribution always? As I understand it, once I reach a large number these intervals will be the same anyway, correct?

You can always use the t-distribution for maximum accuracy, but with enough sessions the contribution will be the same as using the standard normal distribution. Even after only 20 sessions, the t-distribution will only widen your confidence interval by less than 10%.

[southerndog]

n=30 is the rule of thumb that most people use.