Confidence Intervals

[Jerrod Ankenman]

[ QUOTE ]
You apparently do not understand a very important and subtle thing which most people also do not understand. You need to understand this before you put the above in a book, or your book will be seriously flawed.

[/ QUOTE ]

So first of all, I was wrong about the mathematical definition of confidence intervals. Bill, of course, wasn't, and neither is our book because we aren't specifically addressing the type of confidence interval discussed here. I apologize for calling the original post "wrong" or "flawed." In the claims that it makes, it is correct. I stand by my statements that it will often be misleading, especially when it is used by the unsuspecting to answer questions for which it is unsuitable. This is not the fault of the original poster, although it is my general feeling that those who answer questions about inferring information about win rates from observed data (the topic we address in our book) in this manner are doing the askers a disservice.

The "probability" definition of a confidence interval is the most commonly known and understood definition; if one types "define: confidence interval" into Google, as I just did, for example, the first result is "a range of values that has a specified probability of containing the rate or trend....(example deleted)." It is also unhelpful that there is a definition of confidence interval within statistics that is the probability definition that applies to a situation that is not entirely unlike this one to the untrained eye; that is when we know the population mean and want to estimate the outcome of a sample.

<snip>
[ QUOTE ]
[ QUOTE ]
Unfortunately, I don't really have any useful ideas as for incorporating the distribution of all poker player win rates, since a) I don't know it and b) it wouldn't be expressible in closed form.

[/ QUOTE ]
This is precisely the circumstance under which you would use maximum likelihood estimation, and why the use of Bayesian estimation is controversial. The result that you get from Bayesian estimation depends on what prior distribution you assume, and different people will assume different prior distributions. At any rate, the maximum likelihood method which Homer has presented, when understood with the proper mathematical definitions, is mathematically correct.

[/ QUOTE ]
True, understood with the proper mathematical definitions, this statement is mathematically correct. Also, many of the things that I said about this statement were wrong. This doesn't make using this methodology any less misleading to the layperson, however; as evidence of this, I'd enter my own confusion, or if that is not sufficiently compelling, we can poll a hundred smart laypeople and ask them what they think.

So my problem with pointing people to that type of answer to their "win rate certainty" questions is that they are likely to conclude (as I did) that the "confidence interval" reflects a probability that the population mean lies within the interval and use this information to come to consistent overestimation of their particular win rates.
[ QUOTE ]
[ QUOTE ]
I hate criticizing someone else's work without offering a correction or a different methodology; but in this case, I hope it's clear why this method is so biased

[/ QUOTE ]
It is interesting that you used the word "biased" because this has a precise mathematical definition. Bayesian estimates are biased as a result of taking into account our preconceived notions about the prior probability distribution. On the other hand, the maximum likelihood estimate of the win rate is an unbiased estimate because the expected value of the estimate is equal to the true win rate.

[/ QUOTE ]
I'm not a statistician, but Bill claims that this statement is just wrong and that maximum likelihood estimators are not necessarily unbiased. I don't think we need to get into that really. He's at the WSOP now, and it's not the point really.