Pokertracker stats and confidence intervals

[sam h]

I was bored and wondering about the validity of different sample sizes from pokertracker so I decided to calculate a few simple confidence intervals. I think this a good demonstration of the problems with relying on pokertracker too much.

Assume that each individual hand is a case in on our data set and that the observations are independent (not really true, but probably not a huge stretch). For stats like VPIP and PFR, we simply need to calculate confidence intervals using proportions, for a 95% confidence interval - (STAT)+/- (1.96)(SigmaP) where SigmaP = Sqrt(P(1-P)/N)

For a player whose VPIP=30 over 50 hands in pokertracker --> .30 +/- (1.96)(.064) = .30 +/- .127

So for this player we can say with 95% confidence that his true VPIP is between 17 and 43. Doh!

It gets a lot better with 500 hands, where we have a 95% confidence interval of 26-34. Still, however, that is a big difference. With 2000 hands, the interval narrows to 28-32.

The story with PFR is pretty similar. For a PFR of 14, the 95% confidence intervals are:

50 hands = 4.5-23.5
500 hands = 11-17
2000 hands = 12.5-15.5

Of course, we don't need to be 95% confident to draw inferences about how somebody plays. But these numbers were still pretty surprising (and dismaying) to me. I think it speaks to the importance of actual reads based on past actions and continuing to keep more qualitative tabs on players through the notes function. I'm not a big statistics guy, so if I have screwed any of these calculations up please let me know.

[naphand]

Good man.

I made some posts on this a while back and did some preliminary investigations (visiting the probability Forum). I was considering using confidence intervals with substantially less than 95% and was to to look up some z-tables (?) on the web.

The response from this Forum was less than enthusiastic (as you have seen... [img]/images/graemlins/tongue.gif[/img]). It seems people want to believe they can rely on the PT stats far more than is mathematically possible. You will hear endless responses saying hoew the PT stats "help".

What suprises me is that folk will spend hours discussing the finer points of plays, but wont consider the reliability of the PT stats they so dearly love... [img]/images/graemlins/grin.gif[/img]

I think it is worth looing at a sample range of say 50%, 66%, 75% and 90% confidence as these lower ranges may be more appropriate to the kind of probabilities we play with.

The figures you have here really do show just how unreliable (mathemtically at least) PT stats can be. 95% confidence is very high! Of course the PT stats support our visual reads and overall pattern of play from players which does make them more reliable to some extent.

I would like to investigate this a little more, any chance you have access to those z-tables?

[Poldi]

Very interesting.
Surprises me too, thanks for doing this.

[naphand]

The post I made in the probability Forum is here just for completeness. As you can see I am no math Guru... [img]/images/graemlins/ooo.gif[/img]

[afk]

This is fantastic. I want to link this thread every time I see 31/5/1.1 on 37 hands.

[krishanleong]

[ QUOTE ]
This is fantastic. I want to link this thread every time I see 31/5/1.1 on 37 hands.

[/ QUOTE ]

Even 37 hands can be useful. If it's 35/5/1.1 it's not very but if it's 55/1/3 than if can be. You just have to have a good understanding of confidence intervals and player types and stat ranges.

Krishan

[afk]

[ QUOTE ]
[ QUOTE ]
This is fantastic. I want to link this thread every time I see 31/5/1.1 on 37 hands.

[/ QUOTE ]

Even 37 hands can be useful. If it's 35/5/1.1 it's not very but if it's 55/1/3 than if can be. You just have to have a good understanding of confidence intervals and player types and stat ranges.

Krishan

[/ QUOTE ]

Oh I completely agree with you, any kind of numbers have their place as part of a read. I just think that a lot of players (especially newer players) are relying too heavily on PV and GT+ as the only source of info for their reads, which is unfortunate as I believe it will stifle development as a player.

[EvanJC]

[ QUOTE ]
You will hear endless responses saying hoew the PT stats "help".


[/ QUOTE ]

[StellarWind]

[ QUOTE ]
I'm not a big statistics guy, so if I have screwed any of these calculations up please let me know.

[/ QUOTE ]
I didn't check everything but it looks good to me.

[ QUOTE ]
I think it speaks to the importance of actual reads based on past actions and continuing to keep more qualitative tabs on players through the notes function.

[/ QUOTE ]
They have a place. But mathematical debunking should be an equal opportunity exercise. As risky as it is to draw conclusions based on 50 hands of stats, it is even more risky to draw conclusions based on a tiny handful of noteworthy hands. Most players are very inconsistent in how they handle the same situation. It's the mark of a fine player that he minimizes unintentional inconsistency in his play. Most players are extremely prone to Caro's Law of Loose Wiring.

Of course it's virtually never the same situation as the hand from your notes anyway. You may think two situations are comparable but Villain may totally disagree with you.

As perilous as stats are, the small number of reads typically available online are no panacea. Those who have read my analyses for a while may have noticed that I am very skeptical of narrow conclusions about how Villain might play. I assume that much of what I "know" is probably wrong and seek plays that work well over a wide range of conditions. I especially worry about making plays such as tricky folds that are extremely vulnerable to exploitation from a game theory perspective. I consider most of my opponents to be much more random than most players give them credit for.

[sam h]

Here is the Z-score table.

Some quick ones for 50 hand samples of PFR 14 and VPIP 30:

PFR

50% confidence: 11-17
60% confidence: 10-18
70% confidence: 9-18

VPIP

50% confidence: 26-34
60% confidence: 24.5-35.5
70% confidence: 23-37