How many hands do you need to be comfortable with the data?

[sam h]

[ QUOTE ]
Is determining the characteristics of KTo UTG materially more complex than determining the alcohol consumption habits of American Males age 25-31 who drive American cars and have a college education? I posit that the answer is no. This is why, I believe you can begin to ascribe directional significance to PT data after it exceeds around 30-50 data points. Are your results "accurate"? No. Are they "directionally correct?" Perhaps.

[/ QUOTE ]

It is more complex. The crucial question of whether a hand will show a profit relies upon very fine distinctions and the effects of "outlier hands" in which you win a lot of money are too extreme.

Let's say you're trying to evaluate whether you should add KTo to your raising arsenal when UTG. You try it for a while and amass 40 hands doing this. You find that you are losing .1BB/Hand in this situation and decide to stop doing it, as this has now cost you 4BBs. But next time you get dealt it UTG, you raise it anyway out of habit and end up winning a monster pot of 20BBs. Now KTo UTG is a huge winner. So how confident can you be?

[d10]

The analogy was poor. However the point remains valid. You're analyzing your extreme results to decide what kind of sample size you need for your stats to be significant.

Let's say I determine that a given player has exactly a 50% chance of being a winning player (never mind how I made that determination). He comes to me after playing 1,000 hands and shows a negative win rate. These stats are significant. Not significant enough for me to drop his chances of being a winning player down to 0%, in fact no sample size is large enough to be that confident, but I could drop his chances down to 40-something percent. However if I asked someone to pick out their worst session from a larger sample, and that person showed me X amount of hands with a negative win rate, that really wouldn't mean much of anything. I expect that everyone on this board has been through X amount of hands and lost money over that period.

I don't think anyone here will try to tell you that 1,000 hands is enough to get any idea about your win rate. But there's no magic number where a sample turns from worthless to significant. With each additional hand added you can be more confident that the results over your sample match your expected rate. There are plenty of people here who know the exact formula for measuring confidence intervals if that's what you're looking for.

[d10]

I don't know how large the sample needs to be, that's why I specifically avoided making any estimates. I just wanted to point out that the number required to estimate win rate with a specific hand will be much lower than the number required to estimate overall win rate. There are people on this forum who know better than I do the proper methods for measuring confidence intervals, maybe one of them will offer a specific number for X% confidence. Anything I come up with would be purely a guess.

[d10]

Because a sample size is small, you cannot say it tells us nothing at all. You just can't use it to determine anything with absolute certainty.

[King Yao]

I don't know how to quantify this...but I agree with you.

[King Yao]

I agree. Also, you posted some nice follow-up posts in other parts of this thread.

[naphand]

There is a significant amount of discussion on this in the archives.

The general "agreement" was that 100,000 hands was the minimum for providing a reasonably reliable WR (I believe something like +/- 0.5 BB/100). Of course this is ignoring all the other variables such as personal skill development etc. I cannot remember the confidence intervals on this and cannot be bothered to go look it up again - I leave that to those posters determined to do the work. The figures are there.

Confidence intervals are where it is at as far as analysing statistical data goes. Until you are able to understand what they are (and I am using a generic "you" not specifically for OP) and why they are needed this argument will be little more thqan guesswork, and run in ever decreasing circles...

An understanding of the dynamics of numerical analysis is needed to make sense of it. Statistics do not necessarily follow common sense. I am not a statistician, but the principles of mathematical investigations of this type have been drilled into me when studying for a Science degree. I gave up long ago trying to educate forum posters on how to use PT numbers sensibly, they just do not want to know.

This thread has potential as clearly there is interest.

King Yao has not just asked one question here, but many questions about the dynamics and reliability of the stats many of us use. There are simple answers in the archives, but just like in a well-known film, if you do not understand the question the answer will be meaningless.

[naphand]

IMO this is not a valid comparison (the marketing example). As follows:

In your marketing example for "American Males age 25-31 who drive American cars and have a college education" you have already imposed fairly strict criteria on your sample. KTo is the equivalent of perhaps "American Males aged 25-31". Other factors must absolutely be considered when looking at the profitability of a single hand. These are the obvious things such as position, number of players, limit plus a whole bunch of other less significant but influential factors like how many loose-passives, relative position to aggressive players, table image, the reliability of your reads on the other players.

By setting strict criteria, sample size can be reduced. Undoubtedly your "large corporations" have hired the necessary statistical expertise to ensure they can make such decisions based on n criteria and x samples. Unfortunately this has never been done for poker (to my knowledge). Your point is relevant, but unproven and the factors influencing profitability of a particular hand/game have not been defined accurately.

You are correct that grouping similar hands is a way to increase the significance of results. The same exercise applies to position, of course.

[etizzle]

well neither do i exaclty, but basically IMO 50 hands of someone running at 75% tells me a lot about their preflop game.

At least as much as watching 5 hands that they saw the flop with.

[Derek in NYC]

Ok, you raise good points. The cohort "American males 25-31 college educated and American car driving" is likely to be very homogenous. By contrast, "American males 25-31" is a very heterogenous population (including, for example, both PhD grad students at Harvard and convicted murderers serving time in Attica). You are saying, KTo as a population, is more like the latter because its profitability depends on such a broad array of other variables. I buy that.

Now let me ask a question. In SSH, Ed Miller printed a chart showing the relative hand values in BB of various cards. I specifically recall that Ed included a footnote indicating that a particular change in slope for the weakest of hands (and plotted on the lower right hand corner of the graph) was the result of statistical sample anomolies rather than an actual change in profitability.

Does anybody know the empirical basis behind Ed's chart? It certainly was not random-walk, and appeared to be relatively continuous w/out many outliers. (Although I dont know how much best-fit Ed did in plotting this.) Certainly, stepping back, the graph from SSH seemed intuitively correct, in that hands like AA and KK showed a greater profitability than hands like 74o.

Why do we accept that AA is meaningfully positive EV while 74o is significantly negative EV? Ed's graph struck me as directionally correct and non-controversial, and Im willing to bet that it is based on a relatively small dB--certainly nothing like a million hand dB