citanul: a numerical experiment on ROI confidence

[eastbay]

I did this very quickly, so if anyone wants to double check me I'd appreciate it. But here goes anyway...

I took 4 hypothetical players with finish probabilities of:

15 14 13 -> 30% ROI
14 13 12 -> 21%
12 11 10 -> 2.7%
11 10 9 -> -6.4%

Then I asked the question: if each of these players play N tournaments, how often will they see that their ROI falls into the ranges:

< 0
0-10%
10-20%
20-30%
30-40%
>40%

If they take it at face value, they will believe their ROI is in this range (often erroneously).

Some results for various N. Sorry for the imperfect format of this data.

N=50

player 0 (true ROI 30.0)
observed ROI after 50 games:
< 0%: 10.1
0-10%: 10.3
10-20%: 13.1
20-30%: 17.3
30-40%: 16.9
> 40%: 32.2
player 1 (true ROI 20.9)
observed ROI after 50 games:
< 0%: 18.7
0-10%: 14.3
10-20%: 15.4
20-30%: 17.2
30-40%: 14.4
> 40%: 20.0
player 2 (true ROI 2.7)
observed ROI after 50 games:
< 0%: 45.7
0-10%: 17.8
10-20%: 13.6
20-30%: 11.1
30-40%: 6.6
> 40%: 5.2
player 3 (true ROI -6.4)
observed ROI after 50 games:
< 0%: 61.7
0-10%: 15.7
10-20%: 10.2
20-30%: 6.9
30-40%: 3.4
> 40%: 2.1

For N=100:

player 0 (true ROI 30.0)
observed ROI after 100 games:
< 0%: 3.4
0-10%: 8.1
10-20%: 17.2
20-30%: 22.9
30-40%: 21.7
> 40%: 26.6
player 1 (true ROI 20.9)
observed ROI after 100 games:
< 0%: 10.1
0-10%: 15.5
10-20%: 23.7
20-30%: 22.6
30-40%: 16.0
> 40%: 12.1
player 2 (true ROI 2.7)
observed ROI after 100 games:
< 0%: 43.3
0-10%: 24.1
10-20%: 18.8
20-30%: 9.2
30-40%: 3.4
> 40%: 1.1
player 3 (true ROI -6.4)
observed ROI after 100 games:
< 0%: 65.8
0-10%: 19.1
10-20%: 10.4
20-30%: 3.6
30-40%: 0.9
> 40%: 0.2

For N=300:

player 0 (true ROI 30.0)
observed ROI after 300 games:
< 0%: 0.1
0-10%: 1.9
10-20%: 13.5
20-30%: 34.4
30-40%: 34.3
> 40%: 15.8
player 1 (true ROI 20.9)
observed ROI after 300 games:
< 0%: 1.3
0-10%: 11.9
10-20%: 33.8
20-30%: 35.3
30-40%: 15.0
> 40%: 2.6
player 2 (true ROI 2.7)
observed ROI after 300 games:
< 0%: 39.0
0-10%: 40.1
10-20%: 17.7
20-30%: 3.0
30-40%: 0.2
> 40%: 0.0
player 3 (true ROI -6.4)
observed ROI after 300 games:
< 0%: 76.2
0-10%: 20.2
10-20%: 3.3
20-30%: 0.2
30-40%: 0.0
> 40%: 0.0

For N=1000:

player 0 (true ROI 30.0)
observed ROI after 1000 games:
< 0%: 0.0
0-10%: 0.0
10-20%: 3.0
20-30%: 47.6
30-40%: 46.3
> 40%: 3.1
player 1 (true ROI 20.9)
observed ROI after 1000 games:
< 0%: 0.0
0-10%: 1.8
10-20%: 41.2
20-30%: 52.7
30-40%: 4.3
> 40%: 0.0
player 2 (true ROI 2.7)
observed ROI after 1000 games:
< 0%: 30.0
0-10%: 62.2
10-20%: 7.8
20-30%: 0.0
30-40%: 0.0
> 40%: 0.0
player 3 (true ROI -6.4)
observed ROI after 1000 games:
< 0%: 90.2
0-10%: 9.7
10-20%: 0.1
20-30%: 0.0
30-40%: 0.0
> 40%: 0.0

You could probably make up some nice rules of thumb from these numbers.

Cue the usual flame war about random variables...

eastbay

[eastbay]

Anyone who wants to be able to answer q's like this for themselves who doesn't know how to do simple programming in a compiled language like C or C++ is really doing themselves a disservice. This took about 15 minutes to conceive, write, debug, and apply. With just a little more effort and polish, you could use this to find the N necessary to bound your observed ROI say within +/-5% of your true ROI, say 90% of the time.

eastbay

[Slim Pickens]

I'm working on it... geez. FWIW my numbers are coming out about 1% different, but I need to do some checking.

Slim

EDIT: Sample size issues. Sorry. My numbers agree with yours to within that.

[eastbay]

Hang on.. I have better way of quantifying this...

eastbay

[eastbay]

[ QUOTE ]
Hang on.. I have better way of quantifying this...

eastbay

[/ QUOTE ]

I simplified this to counting how often the observed ROI is +/-5% away from the true ROI:

100 games: ~23%
300 games: ~40%
1000 games: ~66%
2000 games: ~82%

These numbers vary a little as a function of true ROI (due to variance differences), but this is good enough for rules of thumb.

Remember, this is only bounding your ROI within a 10% swath, so even for 2000 games when you get there 82-ish% of the time, it's still a fairly rough estimate that you're only making some of the time.

Again, this requires independent verification.

Pretty disheartening, IMO. Fact is, I don't care what my ROI is anymore because I know I can't know. I just try to play each hand the best I know how and let the rest take care of itself.

eastbay

[theredpill5]

Hold on a sec. My password hacker should gain access to Bill Gates' bank account...from there I'm going to wire 100 million dollars into my account.

[Slim Pickens]

[ QUOTE ]
player 0 (true ROI 30.0)
observed ROI after 50 games:
< 0%: 10.1
0-10%: 10.3
10-20%: 13.1
20-30%: 17.3
30-40%: 16.9
> 40%: 32.2


[/ QUOTE ]

20000 trials of 50 tournaments for player 0
< 0%: 9.7
0-10%: 11.0
10-20%: 12.6
20-30%: 17.5
30-40%: 14.6
> 40%: 34.6

Just a sample. It looks similar.

[eastbay]

Can you try the +/-5% test? It makes for a much more digestible result.

eastbay

[Slim Pickens]

[ QUOTE ]
I simplified this to counting how often the observed ROI is +/-5% away from the true ROI:

100 games: ~23%

[/ QUOTE ]

I got 22.9% [img]/images/graemlins/cool.gif[/img]

[Slim Pickens]

[ QUOTE ]
100 games: ~23%
300 games: ~40%
1000 games: ~66%
2000 games: ~82%

[/ QUOTE ]

100 games: 22.9%
300 games: 39.2%
1000 games: 64.9%
2000 games: 81.4%

My sample size is 20k for each. Looks close enough.

Slim