How many hands do you need to be comfortable with the data?

[Zygote]

[ QUOTE ]
Data:

A: 1,900 hands +1.49 BB/100
B: 2,600 hands +3.44 BB/100
C: 1,400 hands -3.29 BB/100
D: 2,200 hands +5.73 BB/100
E: 9,300 hands +4.81 BB/100
F: 1,000 hands -7.43 BB/100

Total: 18,000 hands +3.21 BB/100

(hand totals may be off because I rounded)

A & B are the same limit. C & D are the same limit. E & F are the same limit.

If we believe that 18K hands is enough to say that I am a +1.50 to +4.00 BB/100 player, then what can we say about the results in C & F? If you were shown those results only, and they had 1400 and 1000 hands respectively, you probably would have thought I was a very poor player. But given the full evidence, you probably wouldn’t come to that conclusion.

My overall VPIP is 41.5 (this may be higher than normal because I have played a decent amount of heads-up and 3 player games). But the VPIP for those two losing limits is 34.1 and 34.8, significantly lower. Maybe this shows I was unlucky with good starting hands in these two limits. Or maybe it means I played more 6-handed games in these limits.

The other interesting number is that in F, I had a Won $ at SD % of only 43.9%, while the overall is 51.6%. Maybe this also reflects that I was very unlucky with the board as well in that limit.

If you only saw the results for C & F, maybe you could guess I was unlucky with bad starting cards (but without the overall VPIP, maybe that’s tough to tell), and unlucky in F with Won $ at SD %. However, it would be tough to conclude that with 15K more hands, the overall BB/100 would be such a big difference. So what does this mean? Does this mean that 1K hands is worthless? Maybe.

Can it be then extrapolated that looking at the results 100 hands of KTo UTG in a 6-handed game is even more worthless? If so, how many hands would we need to really tell if that is a good raisable hand or not? 1000? Maybe, but what if KTo was as unlucky as I was in C & F? 5000 hands? Maybe, but I’m not sure I can get that in an entire lifetime of playing every second of my waking hours.

Now what do you do if you see another players stats of 50 hands at -10 BB/100 with a VPIP of 50. Is that worth anything without actually watching him play? Is it possible just watching him play 5 hands will give you more information about the quality of his play than 50 hands worth of stats? Or even 200 hands worth of stats?

Thoughts?

[/ QUOTE ]

Watching someone play 5 hands is worth more than 50 hands of stats and probably a close with 200 hands of stats.

To answer the rest of your questions: plausible confidence intervals

[Justin A]

[ QUOTE ]

Watching someone play 5 hands is worth more than 50 hands of stats and probably a close with 200 hands of stats.

To answer the rest of your questions: plausible confidence intervals


[/ QUOTE ]

Definitely not. What if he's dealt 84o five times in a row?

[Zygote]

[ QUOTE ]
[ QUOTE ]

Watching someone play 5 hands is worth more than 50 hands of stats and probably a close with 200 hands of stats.

To answer the rest of your questions: plausible confidence intervals


[/ QUOTE ]

Definitely not. What if he's dealt 84o five times in a row?

[/ QUOTE ]

You can do virtually nothing based on 50 hands of pure stats with no specific hand reads.

Sometimes that person will play 84o and you will learn something. Regardless, player's will sometimes be distrubted hands that will reveal a lot of information. So is the chance to learn something better than learning virtually nothing? I'd say so..

[d10]

You're selecting the extreme cases out of a sample and ignoring the full sample size. If I had 1,000,000 hands of play and ran bad during 200,000 of them, would you make the conclusion that 200,000 hands is an insignificant sample size? If so you could argue that any sample size is insignificant, and in fact statistics as a whole are insignificant.

On your second point, statistics on individual hands require a much smaller sample to be significant. When looking at absolutely every hand you're dealt, a large amount of those hands are going to be folded before you invest any money into the hand. The hands that have a + or - effect on your earnings will occur infrequently. When looking at individual hands, this is not the case. With stronger hands you might be investing money every single time you're dealt the hand.

[King Yao]

[ QUOTE ]
You're selecting the extreme cases out of a sample and ignoring the full sample size. If I had 1,000,000 hands of play and ran bad during 200,000 of them, would you make the conclusion that 200,000 hands is an insignificant sample size?

[/ QUOTE ]

I disagree with your assessment and I think your analogy is poor. Or maybe its because I didn't make clear what those numbers I posted meant. I didn't take out 1000 hands and put them in F - to show 1000 hands where I lost bigtime. Those were the only 1000 hands at that limit at that site. So its conceivable those 1000 hands were the first 1000 hands I played.

As for running bad over 200K hands with a total of 1,000K hands, I don't think you would see anywhere near the discrepancy in results that 1000 hands would so.

[sam h]

[ QUOTE ]
On your second point, statistics on individual hands require a much smaller sample to be significant. When looking at absolutely every hand you're dealt, a large amount of those hands are going to be folded before you invest any money into the hand. The hands that have a + or - effect on your earnings will occur infrequently. When looking at individual hands, this is not the case. With stronger hands you might be investing money every single time you're dealt the hand.

[/ QUOTE ]

This is a good point, but the question is how large does the sample need to be and is that feasible? If you need several hundred thousand hands to start to get a reasonable assessment of overall winrate and you have a VPIP of 25 then that sample still includes tens of thousands of hands in which you are investing money. Good luck getting tens of thousands of KTo UTG hands.

Plus, arguably a more accurate assessment is needed for the individual hands anyway, since it hardly matters in a practical sense whether you are 3BB/100 or 2.8BB/100 but it matters a lot of KTo UTG is -.05BB/Hand or .05BB/Hand.

[brassnuts]

I think data becomes somewhat reliable around 80k hands.

[etizzle]

the most important thing is that VPIP and pfr% converge much more quickly than BB/100.

BB/100 is a completely useless stat when trying to get a read on someone unless you have at least 20k hands on them.

[mtdoak]

IMHO, the best stat to measure a players skill is winrate. You could show me 2 players with almost identical stats over a massive sample space, but there could be a signifigant gap between win rates. Would this be because one player is 'luckier'? No, it would be because of several factors not measured by stats. Table selection, experience, etc are all factors as well.

For example, i have a 130k hand database, and right now, AQ is more profitable (BB/hand) than AK. Now, that could mean I am misplaying AK, i.e. calling down too often, or it could mean that, even after 130k hands, variance is still a factor.

For your example of KTo, you could have an infinite number of hands, but if you misplayed it every time, you would assume that its a terrible hand based on your stats of it being a big loser, but in truth, you would be the issue.

[Derek in NYC]

[ QUOTE ]
Can it be then extrapolated that looking at the results 100 hands of KTo UTG in a 6-handed game is even more worthless? If so, how many hands would we need to really tell if that is a good raisable hand or not? 1000? Maybe, but what if KTo was as unlucky as I was in C & F? 5000 hands? Maybe, but I’m not sure I can get that in an entire lifetime of playing every second of my waking hours.


[/ QUOTE ]

I think there is a difference between calculating an accurate w/r for a given hand in terms of BB/100, and being directionally correct (e.g., is it +EV or -EV). The former, I happily admit, is hard to do without a huge sample. However I believe the latter is quite possible to do with a much smaller sample.

In quantitative market research, a good rule of thumb is to begin to ascribe statistical significance to results if a sample reaches 30; at a sample size of 50, major business decisions get made based on survey results.

Is determining the characteristics of KTo UTG materially more complex than determining the alcohol consumption habits of American Males age 25-31 who drive American cars and have a college education? I posit that the answer is no. This is why, I believe you can begin to ascribe directional significance to PT data after it exceeds around 30-50 data points. Are your results "accurate"? No. Are they "directionally correct?" Perhaps.

Another way you can get around sample size problems is to group similar categories of hands together. For example, if you were examining the performance of suited aces, you would find that A7s is very similar to A8s. In fact, I suspect that if you ran a hot/cold simulation through Pokerstove, you'd find almost no difference between these hands.

Thus, If you're trying to determine the value of limping with a suited ace-rag UTG, I believe you can fairly take the results for A2s through A6s and treat them all similarly. Your sample size will be 5x as large as for any individual suited A-rag, and not materially inaccurate, I think.

Another thought. Suppose you were to take a player with 100k hands, and plot out the profitability by hand in terms of BB/100. Ed Miller did this in SSH, and he found what seems to be a pretty consistent curve without too many step functions or sawtooths. Now if you look closely, you might find strange anomolies, such as KK being ranked higher than AA. But if you stepped back a level, you would also find, I think that large pocket pairs (AA-JJ) were ranked higher in profits than small pocket pairs (22-66).

I think there is little to be gained by asking questions about specific hands like KTo. It makes more sense to think in terms of hand groupings, much like Ed Miller did when he characterized some hands as being "powerhouses", "speculative hands", etc.

This perhaps is as close as we can get to empirical evidence.