how many SnGs before I have a good idea of my ROI?

[eastbay]

I'm doing an experiment comparing ROI in SnG across buy-ins. How many should I consider a reasonable sample before I can compare the various levels sensibly? Are there some rules of thumb for this?

Thanks,
eastbay

[Prickly Pete]

Bozeman can probably give you a more mathematical answer, but I'd say 200 or 300 will give you a pretty good idea.

[CrisBrown]

Hiya eastbay,

There are formulae for confidence intervals (statistical margins of error) of independent data. For independent data -- e.g.: polling data -- with a very large population, you need a sample size of about 384 to get the confidence interval down to +/- 5%. (See: http://www.surveysystem.com/sscalc.htm )

However, poker results are not independent data. In poker, your opponents get to know you. You get to know them. Your play may improve. A run of bad beats -- or "bad" wins -- may temporarily put you off your best game. And so on.

Poker results are quasi-Markhov Chain data. That is, prior events affect (but do not determine) future results. (See http://www.indiainfoline.com/bisc/jama/jmmm.html ) You need a much larger sample set in order to get the same confidence interval for quasi-Markhov data, typically 3-5x the sample size needed for independent data, depending on the extent of the Markhov effect.

In short, you need about 1200 SNGs to get the margin of error to within 5%.

If you have 300 SNGS at buy-in A and an ROI of 40%, your confidence interval is around 10%. That means you can be 95% sure that your long-term ROI at that buy-in will be between 30% and 50% ... but any number in that range is as likely as any other. You don't have enough data to narrow it farther than that. If you have 300 SNGs at buy-in B and an ROI of 30%, your long-term ROI there could be anywhere from 20%-40%.

So you can't draw any meaningful conclusions about them from only 300 samples each, despite the seemingly obvious greater success at A, because the ranges overlap so much that the difference may be merely a statistical anomaly (you got luckier at A than at B, or were tilting a bit for some of the Bs, or etc.).

Cris

[racingspider]

To a question I haven't thought of yet, but I have been wondering how many would be reliable for feedback on how I am playing. Kudos! I would give you a gold star, but they don't have one, so take a trophy instead! /images/graemlins/cool.gif
Thanks for the info. And while we're on it, how many would be a good set to determine my weaknesses in play (ignoring ROI)? Thanks!

[Bozeman]

The answers given already are all more or less correct, so I will just add that for determination of whether or not you have an edge (and roughly how big it is) the best method is critical examination of your opponents play. There could be some good statistical measure of this, such as average attrition time, number of limps, number of folds headsup, or average allin hand, but these don't seem to be as good as careful study during the play (and probably at least 10-20 sng's). I know that I didn't bother with these until the $33 level, and am lately becoming much more attuned to table makeup.

Craig

[AleoMagus]

I posted a related thread on the probability forum lately and while working throught the numbers, have been surprised. You really do need to play a lot of sngs to have any idea at all with accuracy.

I was ready, in fact, to say that the whole mess was somewhat meaningless because by the time you had aquired a big enough sample, too much could have changed to make the initial results in the series very relevant anymore (especially to the developing player).

This Quasi-Markhov Chain data idea seems to make allowances for this but I still find myself wondering if a player could take it too serious. Over 1200 tournaments, in my experience, positive progression is not gradual, but often marked by 'sudden boosts' which can be tied to identifiable real world experiences. I am talking about things like reading TPFAP or some other book, increasing amount of play, finding a friend to discuss poker with, finding 2+2 boards, the week you read all of Fossilmans posts, etc...

Perhaps (and I don't really know much about stats), the Quasi-Markhov chain data is exactly designed to take this into account, but I'd have a hard time taking my first 300 tourneys where I played over 50 starting hands in the first two rounds and combining it with my last 300, where I played barely 25. Add to this the fact that the first 300 had a ROI significantly less, and I'd be inclined to look past the first 300 altogether.

For an experienced player who plays 300 tourneys each month, it might become entirely feasible to get a fairly static sample of 1200+ tourneys, but for a developing casual player who might only play that many in a year, I'm not so sure.

At any rate, I probably need to know a bit more to say anything meaningful here about stats. Looks like another afternoon af math reading - Now how is Quasi-Markhov spelled again? /images/graemlins/smirk.gif

Regards,
Brad S

[PrayingMantis]

[ QUOTE ]
Over 1200 tournaments, in my experience, positive progression is not gradual, but often marked by 'sudden boosts' which can be tied to identifiable real world experiences

[/ QUOTE ]

Very true. I've played 500+ SNG's, ranging between 2.5 - 22$ buy-in, and I think that any conclusions regarding my ROI, % in the money, etc., are completely undeductable. I feel that every 20-40 games my play goes through a dramatic change, due to analysis of my hands, posting here, reading here, playing 2+2 SNG's, and generally thinking about poker. These are some basic "real world" expiriences that change my game, since I'm still in an early stage, I believe, of my learning process.

However, I will assume that other "real world" expiriences also have impact on the game, even that of pro-players. Two examples:
1. changes in the popularity of poker (there could be a boost of 2-3 month, where poker-rooms are full with fish who watched a certain poker show on TV).
2. other players at a site (especially at higher buy-in's) that are getting famailiar with a player's game, thus making it more difficult for him to win consistently. He will have to adapt to this, and his ROI will skew for a while.


PrayingMantis

[CrisBrown]

Hi Brad,

Yes, for an absolute novice, I agree that by the 1200th sample, the earlier samples will be meaningless data. In the early stages, a developing player will make big jumps in skill. After that, the learning curve flattens, with each new increment of skill requiring more and more study and experience to incorporate fully.

The quasi-Markhov Chain data does take that into account somewhat in requiring a much larger sample set to draw any meaningful statistical conclusions. But at some point you stretch even that beyond its limits, and the early samples need to be filtered out. In that respect, it might be more useful to look at the last 1200 results than at your entire career, as the more recent results more accurately reflect your current expectation.

Regardless, what it means is that you can't draw a whole lot of meaningful data from even 300 SNGs played. There is too much statistical variability involved.

As for drawing data on hands played -- in response to the other question raised -- I think this is less variable. I'd still want 300+ samples for the given hand before I'd be confident in any "exceptional" conclusions. That is, obviously AA is going to be a good hand, and obviously 72o is a bad hand, and I wouldn't need 300+ samples to prove either. But to prove some "exceptional" conclusion, I'd want a lot of data.

E.g.: right now my PokerTracker stats show JJ to have the worst expectation of any hand I play. I've had it twice, and both times it lost huge (once on a set vs. a higher set; once vs. an underpair that hit a straight). But that's only two samples, and I'm going to say that JJ is a sure chip-loser on the basis of only two samples. I'd want 300+ samples before I'd be ready to say that, and I suspect that by the time I have 300+ JJ samples, it'll show a net positive expectation.

Cris