citanul: a numerical experiment on ROI confidence

[eastbay]

I did this very quickly, so if anyone wants to double check me I'd appreciate it. But here goes anyway...

I took 4 hypothetical players with finish probabilities of:

15 14 13 -> 30% ROI
14 13 12 -> 21%
12 11 10 -> 2.7%
11 10 9 -> -6.4%

Then I asked the question: if each of these players play N tournaments, how often will they see that their ROI falls into the ranges:

< 0
0-10%
10-20%
20-30%
30-40%
>40%

If they take it at face value, they will believe their ROI is in this range (often erroneously).

Some results for various N. Sorry for the imperfect format of this data.

N=50

player 0 (true ROI 30.0)
observed ROI after 50 games:
< 0%: 10.1
0-10%: 10.3
10-20%: 13.1
20-30%: 17.3
30-40%: 16.9
> 40%: 32.2
player 1 (true ROI 20.9)
observed ROI after 50 games:
< 0%: 18.7
0-10%: 14.3
10-20%: 15.4
20-30%: 17.2
30-40%: 14.4
> 40%: 20.0
player 2 (true ROI 2.7)
observed ROI after 50 games:
< 0%: 45.7
0-10%: 17.8
10-20%: 13.6
20-30%: 11.1
30-40%: 6.6
> 40%: 5.2
player 3 (true ROI -6.4)
observed ROI after 50 games:
< 0%: 61.7
0-10%: 15.7
10-20%: 10.2
20-30%: 6.9
30-40%: 3.4
> 40%: 2.1

For N=100:

player 0 (true ROI 30.0)
observed ROI after 100 games:
< 0%: 3.4
0-10%: 8.1
10-20%: 17.2
20-30%: 22.9
30-40%: 21.7
> 40%: 26.6
player 1 (true ROI 20.9)
observed ROI after 100 games:
< 0%: 10.1
0-10%: 15.5
10-20%: 23.7
20-30%: 22.6
30-40%: 16.0
> 40%: 12.1
player 2 (true ROI 2.7)
observed ROI after 100 games:
< 0%: 43.3
0-10%: 24.1
10-20%: 18.8
20-30%: 9.2
30-40%: 3.4
> 40%: 1.1
player 3 (true ROI -6.4)
observed ROI after 100 games:
< 0%: 65.8
0-10%: 19.1
10-20%: 10.4
20-30%: 3.6
30-40%: 0.9
> 40%: 0.2

For N=300:

player 0 (true ROI 30.0)
observed ROI after 300 games:
< 0%: 0.1
0-10%: 1.9
10-20%: 13.5
20-30%: 34.4
30-40%: 34.3
> 40%: 15.8
player 1 (true ROI 20.9)
observed ROI after 300 games:
< 0%: 1.3
0-10%: 11.9
10-20%: 33.8
20-30%: 35.3
30-40%: 15.0
> 40%: 2.6
player 2 (true ROI 2.7)
observed ROI after 300 games:
< 0%: 39.0
0-10%: 40.1
10-20%: 17.7
20-30%: 3.0
30-40%: 0.2
> 40%: 0.0
player 3 (true ROI -6.4)
observed ROI after 300 games:
< 0%: 76.2
0-10%: 20.2
10-20%: 3.3
20-30%: 0.2
30-40%: 0.0
> 40%: 0.0

For N=1000:

player 0 (true ROI 30.0)
observed ROI after 1000 games:
< 0%: 0.0
0-10%: 0.0
10-20%: 3.0
20-30%: 47.6
30-40%: 46.3
> 40%: 3.1
player 1 (true ROI 20.9)
observed ROI after 1000 games:
< 0%: 0.0
0-10%: 1.8
10-20%: 41.2
20-30%: 52.7
30-40%: 4.3
> 40%: 0.0
player 2 (true ROI 2.7)
observed ROI after 1000 games:
< 0%: 30.0
0-10%: 62.2
10-20%: 7.8
20-30%: 0.0
30-40%: 0.0
> 40%: 0.0
player 3 (true ROI -6.4)
observed ROI after 1000 games:
< 0%: 90.2
0-10%: 9.7
10-20%: 0.1
20-30%: 0.0
30-40%: 0.0
> 40%: 0.0

You could probably make up some nice rules of thumb from these numbers.

Cue the usual flame war about random variables...

eastbay

[eastbay]

Anyone who wants to be able to answer q's like this for themselves who doesn't know how to do simple programming in a compiled language like C or C++ is really doing themselves a disservice. This took about 15 minutes to conceive, write, debug, and apply. With just a little more effort and polish, you could use this to find the N necessary to bound your observed ROI say within +/-5% of your true ROI, say 90% of the time.

eastbay

[Slim Pickens]

I'm working on it... geez. FWIW my numbers are coming out about 1% different, but I need to do some checking.

Slim

EDIT: Sample size issues. Sorry. My numbers agree with yours to within that.

[eastbay]

Hang on.. I have better way of quantifying this...

eastbay

[eastbay]

[ QUOTE ]
Hang on.. I have better way of quantifying this...

eastbay

[/ QUOTE ]

I simplified this to counting how often the observed ROI is +/-5% away from the true ROI:

100 games: ~23%
300 games: ~40%
1000 games: ~66%
2000 games: ~82%

These numbers vary a little as a function of true ROI (due to variance differences), but this is good enough for rules of thumb.

Remember, this is only bounding your ROI within a 10% swath, so even for 2000 games when you get there 82-ish% of the time, it's still a fairly rough estimate that you're only making some of the time.

Again, this requires independent verification.

Pretty disheartening, IMO. Fact is, I don't care what my ROI is anymore because I know I can't know. I just try to play each hand the best I know how and let the rest take care of itself.

eastbay

[theredpill5]

Hold on a sec. My password hacker should gain access to Bill Gates' bank account...from there I'm going to wire 100 million dollars into my account.

[Slim Pickens]

[ QUOTE ]
player 0 (true ROI 30.0)
observed ROI after 50 games:
< 0%: 10.1
0-10%: 10.3
10-20%: 13.1
20-30%: 17.3
30-40%: 16.9
> 40%: 32.2


[/ QUOTE ]

20000 trials of 50 tournaments for player 0
< 0%: 9.7
0-10%: 11.0
10-20%: 12.6
20-30%: 17.5
30-40%: 14.6
> 40%: 34.6

Just a sample. It looks similar.

[eastbay]

Can you try the +/-5% test? It makes for a much more digestible result.

eastbay

[Slim Pickens]

[ QUOTE ]
I simplified this to counting how often the observed ROI is +/-5% away from the true ROI:

100 games: ~23%

[/ QUOTE ]

I got 22.9% /images/graemlins/cool.gif

[Slim Pickens]

[ QUOTE ]
100 games: ~23%
300 games: ~40%
1000 games: ~66%
2000 games: ~82%

[/ QUOTE ]

100 games: 22.9%
300 games: 39.2%
1000 games: 64.9%
2000 games: 81.4%

My sample size is 20k for each. Looks close enough.

Slim

[eastbay]

[ QUOTE ]
[ QUOTE ]
100 games: ~23%
300 games: ~40%
1000 games: ~66%
2000 games: ~82%

[/ QUOTE ]

100 games: 22.9%
300 games: 39.2%
1000 games: 64.9%
2000 games: 81.4%

My sample size is 20k for each. Looks close enough.

Slim

[/ QUOTE ]

I did 500k, so I think we're good to go. This is "please save for all time, so we have something to point to when people ask this kind of question for the zillionth time" material.

eastbay

[ilya]

Thanks for doing the calcs, e-bay & slim.

[Insty]

Didn't PokerScott already do something like this and post lots of useful tables?

[Apathy]

Could this same thing with confidence intervals be done with ITM as opposed to ROI?

ITM is a much more relavant stastic when measuring your SNG ability at a given level imo.

[Scuba Chuck]

Since I'm always learning, changing my game, and getting better (gulp, I hope), this would seem to me to skew results. These results would suggest that you play the same style the entire sample period.

Just something to add as a sub-note. Thanks, I like the work tho.

[citanul]

[ QUOTE ]
This is "please save for all time, so we have something to point to when people ask this kind of question for the zillionth time" material.

eastbay

[/ QUOTE ]

Bingo, that is the reason I asked for it. As another poster asked, a very similar breakdown for ITM would be great as well. If the guy who posted in the other thread has it perfect, that's cool now, I'll just grab that.

citanul

[nokona13]

Eastbay, are you doing this statistically or empirically? What I mean is, for a given player, say the one with true ROI of 20.9%, are you creating X sets of N games with a loop and a "random" number generator? If so, what are you using for X? Or do you have some way of calculating the variance for each player and then just plugging it into a normal distribution? I only know how to calculate variance given a set of data. Is there some way to convert our knowledge of the players finish distribution into a variance equation for N games played?

[Raiser]

This is empirical. The variance you ask about is a function of the ITM %'s (which, themselves are paramaters of multinomial distributions created by eastbay). From these distributions, and added randomness, simulations can be created to show what would happen over the long haul.

eastbay. Please correct me if this isn't what is going on, but that's what it looks like to me.

[nokona13]

[ QUOTE ]
This is empirical. The variance you ask about is a function of the ITM %'s (which, themselves are paramaters of multinomial distributions created by eastbay). From these distributions, and added randomness, simulations can be created to show what would happen over the long haul.

eastbay. Please correct me if this isn't what is going on, but that's what it looks like to me.

[/ QUOTE ]

Cool, thats' what I was assuming, and what I could probably also do myself in about 20 minutes if I weren't incredibly lazy...

I was wondering if eastbay had devised some ingenious 4 variable formula where X, Y, and Z are the probablitilies of 1st, 2nd, and 3rd, and N is the number of games played, and the equations equals the variance over N games...

[Pokerscott]

[ QUOTE ]
Didn't PokerScott already do something like this and post lots of useful tables?

[/ QUOTE ]

I did and the post is

here (http://forumserver.twoplustwo.com/showflat.php?Cat=&Number=1570111&page=6&view=colla psed&sb=5&o=14&fpart=1)

There may be a problem with the image hosting however. Could someone let me know if they can or can't see the images?

Thanks,

Pokerscott

[pooh74]

[ QUOTE ]
Since I'm always learning, changing my game, and getting better (gulp, I hope), this would seem to me to skew results. These results would suggest that you play the same style the entire sample period.

Just something to add as a sub-note. Thanks, I like the work tho.

[/ QUOTE ]

chuck, see my post in the other thread re: ROI...I like these calcs but it seems it would only be useful to the extent "style of play" is a constant variable...for a bot for instance...my game, I hope too, changes every day, for better or for worse, and this variable could never be accounted for when calculating "true roi" imo. No offense to Eastbay who does us all a service with this work...thx dude!

[jcm4ccc]

this is great stuff.

I wonder if different distributions with the same ROI would come up with different results? Here's what I mean. All 4 of your hypothetical players had similar distributions (1st place a little bit higher than 2nd place which is a little bit higher than 3rd place). What if you have somebody with a higher distribution of 1st places (but with the same ROI)? I would think that the variance would be even higher.

Would you mind running these data? They are kind of ridiculous, but might illustrate that a different distribution with the same ROI can affect the variance.

Player 1: Distribution 15/14/13.
Player 2: Distribution 28.6/0/0

If I did my calculations right, these 2 players have the same ROI. I am betting that the variance of Player 2 is greater, and you would have even less confidence of that player's ROI than player 1 (the confidence interval would be greater). I could be completely wrong, though.

[stupidsucker]

[ QUOTE ]
If I did my calculations right, these 2 players have the same ROI. I am betting that the variance of Player 2 is greater, and you would have even less confidence of that player's ROI than player 1 (the confidence interval would be greater). I could be completely wrong, though.


[/ QUOTE ]

I am curious too. This might be an interesting experiment to see how much ITM% directly affects variance.

[Raiser]

[ QUOTE ]
[ QUOTE ]
If I did my calculations right, these 2 players have the same ROI. I am betting that the variance of Player 2 is greater, and you would have even less confidence of that player's ROI than player 1 (the confidence interval would be greater). I could be completely wrong, though.


[/ QUOTE ]

I am curious too. This might be an interesting experiment to see how much ITM% directly affects variance.

[/ QUOTE ]

jcm's guess is right. The variance, with respect to ROI, for the 28.6/0/0 player is higher than the 15/14/13 player.

[Slim Pickens]

This is from my baby sim, which benchmarked well against eastbay's. The sample size is now 25k (his was 250k).

input ROI=30%
input finish distribution: 26.8% 1st, 0% 2nd, 0% 3rd

# of tournaments:% of the time ROI is within +-5% of input value

100: 18%
500: 42%
1000: 56%
2000: 72%

Slim

[bball904]

[ QUOTE ]
This is from my baby sim, which benchmarked well against eastbay's. The sample size is now 25k (his was 250k).

input ROI=30%
input finish distribution: 26.8% 1st, 0% 2nd, 0% 3rd

# of tournaments:% of the time ROI is within +-5% of input value

100: 18%
500: 42%
1000: 56%
2000: 72%

Slim

[/ QUOTE ]

Finally some analysis of value. This comes very close to my 27/2/3 distribution over my last 1500 55's. Thanks for the "analysis".

Thanks also to Scuba and pooh for adding some common sense posts to this thread.

[Slim Pickens]

Aw... my first bball troll/flame. This one's going on my fridge.

[Barrett's Last Privateer]

Guys,

Thanks heaps for taking the time to do this.

Particulalry liking the switch around to 'for a given observed ROI I'm 90% confident my true ROI is between x and y.'

Obviously knowing the potential range of observed ROIs for a given true ROI is also interesting, but as it's the observed ROI which we actually see, and the true ROI about which we want to make inferences (albeit recognising the assumptions we have made), looking at things from this way is particulalry useful.

BLP

[ReDeYES88]

[ QUOTE ]
Since I'm always learning, changing my game, and getting better (gulp, I hope), this would seem to me to skew results. These results would suggest that you play the same style the entire sample period.

Just something to add as a sub-note. Thanks, I like the work tho.

[/ QUOTE ]

I was just formulating a similar post in my head as I scrolled down and came across yours. Good point.