[BruceZ]

We frequently see posts on the low-limit forum and other places in which the poster says they have calculated a win rate of 5 BB/hr over 100 hours (or whatever), and everyone laughs and says that 100 hours is not nearly long enough to calculate valid statistics.

You can make confidence intervals after a small number of hours, as long as you keep in mind that they only apply to the conditions that were in effect when you played. It is likely that 100 hours may not be a large enough sample to adaquately represent the future if conditions are likely to change, for example, if you have a different mix of opponents in the future who play significantly differently than the opponents over the 100 hours. Still you can construct confidence intervals for the conditions which you did play against during those 100 hours.


So we can say with at least 90% confidence that this player is at least break-even. Is there an error in this argument?

Firstly, if this were the 90% confidence interval, then there would be a 95% confidence that the player is at least break-even since for this we want the 1-sided confidence interval, and only 5% falls below 0. Secondly, your calculations would be perfect if your estimate of the standard deviation were perfect. The SD converges fast enough that this is usually a minor factor, but for only 6 sessions the uncertainty in the SD becomes significant. To correct for this, you need to use the t-distribution. First multiply your SD by a factor of sqrt(6/5). Then for a 90% confidence interval, you would use the inverse t-distribution with 5 degrees of freedom. In Excel this is TINV(0.1,5) = 2.015. Use this instead of 1.64. These two corrections will cause your actual 90% confidence interval to widen by 35%, to +/- $47, instead of +/- $35. So it would go from -$11 to $83. 0 would lie at the edge of the 80% confidence interval, so there would be an 85% confidence of being a winning player.