Just What IS Statistically Significant?

[Festus22]

I've read ad nauseum here that 1000 hands is a statistically insignificant sample size to draw valid conclusions regarding one's play. So, just how big does the sample size need to be to conclude some kind of result at let's say 90% confidence? Or 95% How is this derived?

[Warren Whitmore]

P(X-1.96(sigma/square root of n)<u<X+1.96(sigma/square root of n)= 0.95 or 95%

[Bozeman]

What conclusions are you trying to draw? If you have been dealt AKs 1000 times, you can make some reasonable predictions, except for the caveat that the game was slightly different each time.

Craig

[Festus22]

My win rate or if I have improved. Could you apply an F or t test between two blocks of hands to see if you've improved your win rate or maybe lowered your variance in a statistically significant way? I think that really is the basis of my original question in that how big should the sample size be to get a valid conclusion whether one has improved or not.

[bigpooch]

I think for a game like LHE, for the statistics to be
meaningful, a player may have to keep track of a lot of data
(much more than most people are willing to record!). A good
suggestion (though not many players would want to do this!)
would be to record your results for each orbit, just before
you are going to post your BB that round. Online is much
easier even if you are playing multiple games: keep a
program like wordpad open and just type in your chip total
for each of the tables (often you'll be utg and mucking).

Suppose a hypothetical player has a win rate of 0.3 BB per
orbit (or per ten hands for simplicity; say this player
only plays ring games) and has a SD of about 7 BB per orbit.
The above amounts to the usual B&M statistics for a typical
solid winning player in medium limits. Most statisticians
use the normal approximation for very good theoretical
reasons (e.g., central limit theorems) but for a 95%
confidence interval, the results for this player for one
orbit should fall within +/- 1.96 x SD or about 14 BB with
a probability of 95%. By the way, that 1.96 is just the
z-value so that the tails of the normal distribution only
add up to 5% in area so that the central 95% of the results
are only considered (for more details, consult a statistics
text!).

In reality, of course, the results after each orbit would be
slightly skewed to the right (it's much more likely this
player will win a bunch of chips than lose an equivalent
amount!) but this is just an approximation anyway. As the
number of orbits increase, this 95% confidence interval of
the raw results will clearly increase in size. By how much?
By the square root of the number of orbits.

For example, suppose this player played 100 orbits. The
95% confidence interval would now be

100 x (0.3) +/- 1.96 x 7 x sqrt(100) or

30 +/- 137.2 (in BBs)

Similarly, for 10000 orbits, it would be

3000 +/- 1372.

Thus, this player could argue that his win rate would be
determined within 0.14 BB per orbit with 95% confidence
after playing 10000 orbits. This would be true of any other
player with the same SD as this is the determining factor
for how wide the confidence interval is.

But notice that fraction 1372/3000 is not that small which
means that the hourly rate cannot really be determined with
much accuracy. Besides, game conditions change, the player
may improve (or get worse!), and he may play short-handed,
etc. For someone who is not sure how much he should expect
to make playing, then I would encourage him to keep detailed
records of all results. 10000 orbits is easily obtained by
playing one table online in a year. For those playing two
or more games, the data would obviously be more significant.

Of course, we didn't have to keep track of data for orbits;
it could have been for hours, 15 minute time intervals or
every 100 hands. Usually people keep their personal data
with hours in mind but this seems quite artificial as the
game moves not according to the clock!