Sample Sizes

[Dav123]

I wasn't going to post on this board, and I'm still sure I will refrain from any strategy related posts, but something that comes up constantly really bothers me. You guys talk about playing 100 SNGs or 200 SNGs and your results. Then you want to figure out whether or not to move up or change or game based on these results.

THESE SAMPLE SIZES ARE NO WHERE NEAR LARGE ENOUGH FOR ANY KIND OF MEANINGFUL ANALYSIS.

There is a lot of luck in poker, and chances are that any patterns, % in the money, and ROI are more of a factor of luck than anything within your control after 100 SNGs. Sure the guy who has an ROI of 40 is prob a better player than a guy with an ROI of 12, but he certainly doesn't have to be. Thats another point, ROI is garbage when it comes to poker. When you play poker, the most important thing you invest is not your money, but your time. If you are a winning player, $/hour is a much better measurement, provided you keep your risk at an acceptable level (don't play beyond your bankroll)

Don't believe me about the sample sizes? I have more than 3000 SNG's tracked in Poker Tracker since March when I found their profitability. In my 1st 400 (yes, 400) $20 SNGs I was averaging a profit of $10 per tournament. Over the course of another 300, I was averaging a $1 loss per tournament. After 2100 $20 SNGs? Profit of $4.70 per tournament. So 400 wasn't nearly enough to figure out how much I could actually make in them. Is 3000 enough? Get this. I am making 3 times as much per hand with AKo vs. AKs. Of course I play them exactly the same. 65s is a solid winner for me. 54s is my biggest loser. More than 72o and every other hand.

Just some food for thought.

[SparkMan]

Your $4.17 profit per tournament sounds a lot more realistic than some of the ROI claims I've seen posted. At least from a long term perspective.

[TJD]

Better mathematicians by me may post in reply here but my initial thoughts are.

1) Hand analysis needs many more hands for things to even out hence the discrepancies.

2) It is true that we do not KNOW what our real long term performance will be when all good and bad luck have evened out but statistics does not claim to do that. All it tries to do is decide based on the evidence, how likely it is that the results we get could have been acheived by chance.

Let's assume that after 200 SnGs you are ITM 45%. I think we can use the normal approximation to the binomial distrubution to determine the likelihood that our ITM% is to be expected based on a hypothesis as to what our "real" ITM is.

If we assume that our real ITM is 30% then the SD of the normal distribution would be

SQRT{0.3 x 200 (1 - 0.3)}= 6.48

Since we have produced a result 15% above that, we are 2.3 SD from our hypothesis which leaves about a 1% chance that the 45% result occurred by chance when our real performance is only 30%. Hence I would feel comfortable that my performance was better than 30%. I could be less confident that my true performance was better than 35% and even less so about it being better than 40%.

This is what statistics is trying to do.

It is a lng time since I studied statistics, so I may have made an error,(probably with 1 or 2 tail tests) but the principle is correct:-

We can estimate how likley the result is to be believed.

We play poker; it is uncertain; all things are uncertain but if you can advise a developing player that it is 99% likley that their good results are NOT just due to good luck then that is probably good enough odds to take the risk of moving up on.

Trevor

[joker1976]

I have over 7000 SnGs so far. 3000 at $10-$30, 4000 at $50, and some at > $50. Experience tell me that even 500 SnGs doesn't mean much. I have 500 games rocking at 50% ROI, as well as 500 games getting 10% ROI. When I reached 6400 SnGs, my long term average was still 31%, now it dropped to 26%. Because the players change everyday, this number isn't very useful, the only thing it tells me is that I used to be a winner at rate around 26%-31%.

What about my next 500 games? I won't dare to predict the theoretical result with any accuracy, because even a small change of average player skill can make a big difference. If a person receives 30% ROI, his $win/game is actually 140% of the buy in. If the game gets 10% more difficult to win money, your $win/game is then 140%*0.9 = 126% of the buy in, and ROI drops to 16%, not 30%*0.9 = 27%. Predicting future return based on past stats is very wrong unless you can play minimum 30 SnGs each day.

[joker1976]

I don't know much about stats, but I feel that it can't be right. I have received over 40% and less than 10% ROI many times each 200 SnGs (about 1/3 of all games). My average return is 26% so far. So either I have received some impossible swings or your calculation is not correct or I have misread your post. I don't know much about stats calculation, can anyone else confirm it?

[Dav123]

ROI does not fit into SQRT. He is using ITM% (of course almost meaningless when it comes to SNG results) which also is a very questionable fit to SQRT.

[stripsqueez]

apparently you only know your win rate when you die so i've decided not to bother with it anymore

what win rate stats tell me is that i have to constantly try and get better

stripsqueez - chickenhawk

[joker1976]

I just checked my result, they range from 34% - 43%, averages 40%.

[Jason Strasser]

I think most people who do the 100 SNG thingy are doing it for more recreational purposes. It's so that people can compare short term stats and yadda-yadda about them.

I do hope no one is moving up or down in SNGs based on a small sample size...

[for teh win]

"Does not fit into SQRT"? In e.g. finance it's not a problem to use statistics on returns or valuations of a position or portfolio. Personally I would prefer running a regression on my SNG bankroll. But should we assume poker is governed by a different set of mathematical laws compared to other "sciences"? Why do we need bigger sample sizes in poker than elsewhere?

I can't understand this constant whining about sample sizes in the forums. A sample of 100 or 200 SNGs is perfectly fine. You just have to make sure the sample is random or at least representative. (As opposed to e.g. using the results to predict future success at a different casino or failing to account for seasonality effects like day of week.) Usually, though, the sample is representative.

Can a sample of 200 SNGs with outstanding results be a lucky streak? Of course it can, and the confidence interval tells you exactly how (un)likely that is. A confidence interval is a confidence interval, and it says what it says, no more no less. It doesn't claim to give a "true" answer. But it sure beats gut feeling in telling you where you stand. And it beats ill advice too.

Is a sample of 1000 better than 200? Sure, but large samples are problematic too because it is hard to account for factors like skill increases and changes at your usual tables (stiffening competition over time or whatever). Hence the latest entries in your sample will always be the most valid. In other words, a small subsample of your total sample is the best predictor for the immediate future.

What about using statistics from your past results to predict future success? Can you do that at all? It's an inference. But everyone who ever studied statistics knows that already. Still, if you need help deciding whether to move up in limits or not now that your bankroll admits it, where do you turn to? To your gut feeling or the howling crowd that demands you have 10,000 logged games before you open your mouth in here? Or do you turn to the best pointer you can get - statistics? It will be an inference and thus it can't be trusted as a "true" answer and everyone knows that, but it's something at least.

Apart from some really nice discussions on probabilistics, I often find the forum attitude towards statistics similar to people trying to reinvent the wheel, then destroy it because it wasn't a perfect circle. But it's not supposed to be a perfect circle! It's supposed to be a wheel, so that you can move forward.

"In the long run we're all dead."
- J M Keynes

"In the long run PP will have shut down."
- anonymous