Confidence Intervals

[Dazarath]

This isn't really probability, but I figure if that since a lot of mathematics related questions are posted in this forum, this is the best place to ask. Given my current winrate and standard deviation per 100 hands from Poker Tracker, how do I go about calculating confidence intervals for my true winrate? My thanks to anyone who can help me out.

[gaming_mouse]

SD = your sample standard deviation (in BB/100)
sample_size = (number of hands)/100
sample_winrate = your sample winrate

you standard error (SE) is:

SD/sqrt(sample_size)

A 95% CI for your true winrate is sample_winrate +- 2*SE

HTH,
gm

[BruceZ]

[ QUOTE ]
This isn't really probability, but I figure if that since a lot of mathematics related questions are posted in this forum, this is the best place to ask. Given my current winrate and standard deviation per 100 hands from Poker Tracker, how do I go about calculating confidence intervals for my true winrate? My thanks to anyone who can help me out.

[/ QUOTE ]

Try this post (http://forumserver.twoplustwo.com/showthreaded.php?Cat=&Board=probability&Number=192 2208&Forum=,All_Forums,&Words=&Searchpage=0&Limit= 25&Main=1921919&Search=true&where=&Name=197&datera nge=&newerval=&newertype=&olderval=&oldertype=&bod yprev=#Post1922208).

[Orpheus]

This (http://forumserver.twoplustwo.com/showflat.php?Cat=&Board=inet&Number=1342415&Forum= All_Forums&Words=%2B%20confidence%20%2Binterval%20 %2Bspreadsheet&Searchpage=0&Limit=99&Main=1342415& Search=true&where=bodysub&Name=&daterange=1&newerv al=0&newertype=w&olderval=1&oldertype=m&bodyprev=# Post1342415) is the thread I refer people to. It includes a small spreadsheet formula to stick in the corner of your overall poker spreadsheet [but read down to the corrections]

[Dazarath]

Ok, thanks for the responses guy, I'll read over everything right now.

[Jerrod Ankenman]

[ QUOTE ]
This (http://forumserver.twoplustwo.com/showflat.php?Cat=&Board=inet&Number=1342415&Forum= All_Forums&Words=%2B%20confidence%20%2Binterval%20 %2Bspreadsheet&Searchpage=0&Limit=99&Main=1342415& Search=true&where=bodysub&Name=&daterange=1&newerv al=0&newertype=w&olderval=1&oldertype=m&bodyprev=# Post1342415) is the thread I refer people to. It includes a small spreadsheet formula to stick in the corner of your overall poker spreadsheet [but read down to the corrections]

[/ QUOTE ]

By the way, the post that BruceZ linked to makes reasonable statements. The post that is linked to in this post contains a lot of statements of dubious value. In fact, repeatedly utilizing the methodology there will lead to making false statements with probability 1.

Jerrod Ankenman

[uuDevil]

[ QUOTE ]
[ QUOTE ]
This (http://forumserver.twoplustwo.com/showflat.php?Cat=&Board=inet&Number=1342415&Forum= All_Forums&Words=%2B%20confidence%20%2Binterval%20 %2Bspreadsheet&Searchpage=0&Limit=99&Main=1342415& Search=true&where=bodysub&Name=&daterange=1&newerv al=0&newertype=w&olderval=1&oldertype=m&bodyprev=# Post1342415) is the thread I refer people to. It includes a small spreadsheet formula to stick in the corner of your overall poker spreadsheet [but read down to the corrections]

[/ QUOTE ]

By the way, the post that BruceZ linked to makes reasonable statements. The post that is linked to in this post contains a lot of statements of dubious value. In fact, repeatedly utilizing the methodology there will lead to making false statements with probability 1.

Jerrod Ankenman

[/ QUOTE ]

That post has been referred to quite often on these forums. Would you (or perhaps BruceZ, pzhon, or other resident mathematician) please specify what these dubious statements are?

By the way, when does your book come out?

[Jerrod Ankenman]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
This (http://forumserver.twoplustwo.com/showflat.php?Cat=&Board=inet&Number=1342415&Forum= All_Forums&Words=%2B%20confidence%20%2Binterval%20 %2Bspreadsheet&Searchpage=0&Limit=99&Main=1342415& Search=true&where=bodysub&Name=&daterange=1&newerv al=0&newertype=w&olderval=1&oldertype=m&bodyprev=# Post1342415) is the thread I refer people to. It includes a small spreadsheet formula to stick in the corner of your overall poker spreadsheet [but read down to the corrections]

[/ QUOTE ]

By the way, the post that BruceZ linked to makes reasonable statements. The post that is linked to in this post contains a lot of statements of dubious value. In fact, repeatedly utilizing the methodology there will lead to making false statements with probability 1.

Jerrod Ankenman

[/ QUOTE ]

That post has been referred to quite often on these forums. Would you (or perhaps BruceZ, pzhon, or other resident mathematician) please specify what these dubious statements are?

By the way, when does your book come out?

[/ QUOTE ]

Our book discusses this topic a little, but it's one thing to criticize a flawed methodology and another to substitute a different one. Basically, the problem at hand is inferring a win rate from observed data.

If you KNOW the win rate (because Allah told you or something), then you can create a confidence interval for a sample, as long as your sample is big enough to satisfy the Central Limit Theorem (which it normally will be for poker-size HAND samples; tournaments are another thing entirely).

But that's not what we're doing here. Here we want to take an observed (sample) win rate and turn it into a true (population) win rate. Now in a lot of fields, we sorta just use x-bar (the sample mean) as the best estimator of population the mean and s, the sample standard deviation, as the best estimator of the population standard deviation, because we don't have any other data to work with.

But in poker, we do have a lot of other data that tells us something about the a priori distribution of all poker players. Say we have a player who has won 6 bb/100 hands over 10,000 hands (600 bb), with a variance of 4bb^2/h. Now using these parameters as best estimators, we get that the standard error of a 10,000 hand sample is 200 bb. So using a normal distribution with stddev 200 bb and mean 600, we are led to the following statements:

It is 95% likely that this player's win rate is between 2bb/100 and 10bb/100.

It is equally likely that this player's win rate is above 6bb/100 as it is that it is below 6bb/100.

It is equally likely that this player's win rate is 10bb/h as it is that the player's rate is 2bb/h.

Now I ask the following question; is it more likely that this player is a decent winner having a result a couple standard deviations to the right of his mean? Or is it more likely that he's really a player with a rate that is some multiple of the highest reliable rates reported. Of course the former. Bayes' theorem allows us to incorporate information about the distribution of all poker players into our analysis. What this leads to is that the formula in that post will likely overestimate the likelihood that a player is a winner (for the players who will actually use it), and also overestimate the probabilty that a player has a very high win rate. This is especially true because players who are trying to prove that they are winners after a reasonably small sample will normally be experiencing highly positive results relative to their mean.

Unfortunately, I don't really have any useful ideas as for incorporating the distribution of all poker player win rates, since a) I don't know it and b) it wouldn't be expressible in closed form, probably, anyway, and so wouldn't be that useful for finding a formula to answer the question "how many hands do I need to be a winning player?"

I hate criticizing someone else's work without offering a correction or a different methodology; but in this case, I hope it's clear why this method is so biased.

We will be wrapping up the first complete draft of our book in the next two weeks or so; then there's some sort of long editing process, and so on. But the book should be in stores late this fall.

Jerrod Ankenman

[BruceZ]

You apparently do not understand a very important and subtle thing which most people also do not understand. You need to understand this before you put the above in a book, or your book will be seriously flawed.

Homer's method of analysis is fundamentally valid. Your interpretation of what his analysis says is incorrect.

[ QUOTE ]
But in poker, we do have a lot of other data that tells us something about the a priori distribution of all poker players. Say we have a player who has won 6 bb/100 hands over 10,000 hands (600 bb), with a variance of 4bb^2/h. Now using these parameters as best estimators, we get that the standard error of a 10,000 hand sample is 200 bb. So using a normal distribution with stddev 200 bb and mean 600, we are led to the following statements:

It is 95% likely that this player's win rate is between 2bb/100 and 10bb/100.

It is equally likely that this player's win rate is above 6bb/100 as it is that it is below 6bb/100.

It is equally likely that this player's win rate is 10bb/h as it is that the player's rate is 2bb/h.

[/ QUOTE ]

These statements are all incorrect, and they do NOT follow from the analysis of confidence intervals that Homer has provided. This statement from Homer's post:

[ QUOTE ]
"I am 95% confident that my true win rate is between -$24 and $102 per table-hr."


[/ QUOTE ]

IS correct. The difference is the word "confident". Confidence has a mathematically defined meaning distinct from "likely" or "probability". Homer's statement does not mean that there is a 95% probability that his true win rate is in this range. It means that IF the true win rate were above this range, then the probability of observing a win rate equal to or less than the observed win rate would be less than 2.5%, and IF the true win rate were below this range, then the probability of observing a win rate equal to or greater than the observed win rate would also be less than 2.5%.

Another way to look at it is to note that if a large number of people constructed their 95% confidence interval in this manner, then 95% of the people would have their true win rate fall in this interval. This is because before they played any hands, there was a 95% probability that their observed win rate would fall within +/- 2 standard deviations of their true win rate. Conversely then, there was a 95% probability that their true win rate would lie within +/- 2 standard deviations of their observed win rate. After they play the hands, they then construct a +/- 2 standard deviation confidence interval around their observed win rate. However, it is no longer correct to say that their true win rate has a 95% probability of lying within this interval because their true win rate and their observed win rate are not probability distributions; they are fixed numbers. Thus we characterize the situation by saying that there is a 95% confidence that the true win rate lies within this interval.

Note that the center of this interval is the observed win rate, or the "sample mean", and this is termed the "maximum likelihood estimate" of the mean, assuming that we have enough samples that the sample mean is well-approximated by a normal distribution. This term "maximum likelihood estimate" does not mean that this is the most likely value of the mean. It means that this is the value of the mean which maximizes the likelihood of the observed win rate.

This is a completely accurate and mathematically correct statement. It is the only type of statement that can be made when using this type of statistics, called "maximum likelihood estimation". The type of analysis you are proposing, whereby we take into account our prior knowledge about the population distribution and actually compute a probability distribution for our win rate, is called "Bayesian estimation".

[ QUOTE ]
Here we want to take an observed (sample) win rate and turn it into a true (population) win rate.

[/ QUOTE ]

That is NOT what we are doing with maximum likelihood estimation, which Homer is performing. This is what we would do with Bayesian estimation.

Now the argument that Bayesian estimation is the only valid form of estimation that should be used is an argument that some statisticians make. It is controversial for the very reason that you state:

[ QUOTE ]
Unfortunately, I don't really have any useful ideas as for incorporating the distribution of all poker player win rates, since a) I don't know it and b) it wouldn't be expressible in closed form.

[/ QUOTE ]

This is precisely the circumstance under which you would use maximum likelihood estimation, and why the use of Bayesian estimation is controversial. The result that you get from Bayesian estimation depends on what prior distribution you assume, and different people will assume different prior distributions. At any rate, the maximum likelihood method which Homer has presented, when understood with the proper mathematical definitions, is mathematically correct.

[ QUOTE ]
I hate criticizing someone else's work without offering a correction or a different methodology; but in this case, I hope it's clear why this method is so biased

[/ QUOTE ]

It is interesting that you used the word "biased" because this has a precise mathematical definition. Bayesian estimates are biased as a result of taking into account our preconceived notions about the prior probability distribution. On the other hand, the maximum likelihood estimate of the win rate is an unbiased estimate because the expected value of the estimate is equal to the true win rate.

[Siegmund]

[ QUOTE ]
On the other hand, the maximum likelihood estimate of the win rate is an unbiased estimate because the expected value of the estimate is equal to the true win rate.

[/ QUOTE ]

No time like the present to remind y'all that MLEs don't come with an automatic guarantee of being unbiased either (and for non-symmetric distributions they usually aren't), only that they asymptotically approach the unbiased value in an efficient fashion.

Not commenting on the actual question at hand, except to affirm the general idea that there are a few different types of calculations you can do, each of which requires a certain set of assumptions that are often glossed over.

[BruceZ]

Sorry, but I muffed the most important sentence of this explanation. I already edited the above post with the following correction to this paragraph:

[ QUOTE ]
Confidence has a mathematically defined meaning distinct from "likely" or "probability". Homer's statement does not mean that there is a 95% probability that his true win rate is in this range. It means that IF the true win rate were outside of this range, there would be less than a 5% probability that his observed results would occur.

[/ QUOTE ]

That would only make sense if the confidence interval were 1-sided, as would be the case if we testing the hypothesis that our win rate is strictly above or below a certain value. Even then, "observed results" would have to be replaced by "observed results or worse" or "observed results or better". In this case we were talking about a 2-sided confidence interval. This paragraph has been replaced with the following paragraphs:

Confidence has a mathematically defined meaning distinct from "likely" or "probability". Homer's statement does not mean that there is a 95% probability that his true win rate is in this range. It means that IF the true win rate were above this range, then the probability of observing a win rate equal to or less than the observed win rate would be less than 2.5%, and IF the true win rate were below this range, then the probability of observing a win rate equal to or greater than the observed win rate would also be less than 2.5%.

Another way to look at it is to note that if a large number of people constructed their 95% confidence interval in this manner, then 95% of the people would have their true win rate fall in this interval. This is because before they played any hands, there was a 95% probability that their observed win rate would fall within +/- 2 standard deviations of their true win rate. Conversely then, there was a 95% probability that their true win rate would lie within +/- 2 standard deviations of their observed win rate. After they play the hands, they then construct a +/- 2 standard deviation confidence interval around their observed win rate. However, it is no longer correct to say that their true win rate has a 95% probability of lying within this interval because their true win rate and their observed win rate are not probability distributions; they are fixed numbers. Thus we characterize the situation by saying that there is a 95% confidence that the true win rate lies within this interval.

Note that the center of this interval is the observed win rate, or the "sample mean", and this is termed the "maximum likelihood estimate" of the mean, assuming that we have enough samples that the sample mean is well-approximated by a normal distribution. This term "maximum likelihood estimate" does not mean that this is the most likely value of the mean. It means that this is the value of the mean which maximizes the likelihood of the observed win rate.

[BruceZ]

[ QUOTE ]
[ QUOTE ]
On the other hand, the maximum likelihood estimate of the win rate is an unbiased estimate because the expected value of the estimate is equal to the true win rate.

[/ QUOTE ]

No time like the present to remind y'all that MLEs don't come with an automatic guarantee of being unbiased either (and for non-symmetric distributions they usually aren't), only that they asymptotically approach the unbiased value in an efficient fashion.

[/ QUOTE ]

The sample mean is always an unbiased estimate of the mean of any distribution by definition. By "maximum likelihood estimate of the win rate", I was actually referring to the sample mean which we use as an estimate of the win rate when performing a maximum likelihood analysis. The central limit theorem says that for a sufficient number of hands, the distribution of the observed win rate, or sample mean, will be well-approximated by a normal distribution with a mean equal to the true win rate. We then identify the sample mean as the MLE of the mean of this normal distribution, and this allows us to use normal confidence intervals. I realize that this isn't the same as saying that we have an MLE for the win rate which is unbiased unless the sample mean is exactly normally distributed, but it is an unbiased estimate of the win rate, and it can be made arbitrarily close to the MLE of the win rate for a sufficient number of samples.

[Jerrod Ankenman]

[ QUOTE ]
You apparently do not understand a very important and subtle thing which most people also do not understand. You need to understand this before you put the above in a book, or your book will be seriously flawed.

[/ QUOTE ]

So first of all, I was wrong about the mathematical definition of confidence intervals. Bill, of course, wasn't, and neither is our book because we aren't specifically addressing the type of confidence interval discussed here. I apologize for calling the original post "wrong" or "flawed." In the claims that it makes, it is correct. I stand by my statements that it will often be misleading, especially when it is used by the unsuspecting to answer questions for which it is unsuitable. This is not the fault of the original poster, although it is my general feeling that those who answer questions about inferring information about win rates from observed data (the topic we address in our book) in this manner are doing the askers a disservice.

The "probability" definition of a confidence interval is the most commonly known and understood definition; if one types "define: confidence interval" into Google, as I just did, for example, the first result is "a range of values that has a specified probability of containing the rate or trend....(example deleted)." It is also unhelpful that there is a definition of confidence interval within statistics that is the probability definition that applies to a situation that is not entirely unlike this one to the untrained eye; that is when we know the population mean and want to estimate the outcome of a sample.

<snip>
[ QUOTE ]
[ QUOTE ]
Unfortunately, I don't really have any useful ideas as for incorporating the distribution of all poker player win rates, since a) I don't know it and b) it wouldn't be expressible in closed form.

[/ QUOTE ]
This is precisely the circumstance under which you would use maximum likelihood estimation, and why the use of Bayesian estimation is controversial. The result that you get from Bayesian estimation depends on what prior distribution you assume, and different people will assume different prior distributions. At any rate, the maximum likelihood method which Homer has presented, when understood with the proper mathematical definitions, is mathematically correct.

[/ QUOTE ]
True, understood with the proper mathematical definitions, this statement is mathematically correct. Also, many of the things that I said about this statement were wrong. This doesn't make using this methodology any less misleading to the layperson, however; as evidence of this, I'd enter my own confusion, or if that is not sufficiently compelling, we can poll a hundred smart laypeople and ask them what they think.

So my problem with pointing people to that type of answer to their "win rate certainty" questions is that they are likely to conclude (as I did) that the "confidence interval" reflects a probability that the population mean lies within the interval and use this information to come to consistent overestimation of their particular win rates.
[ QUOTE ]
[ QUOTE ]
I hate criticizing someone else's work without offering a correction or a different methodology; but in this case, I hope it's clear why this method is so biased

[/ QUOTE ]
It is interesting that you used the word "biased" because this has a precise mathematical definition. Bayesian estimates are biased as a result of taking into account our preconceived notions about the prior probability distribution. On the other hand, the maximum likelihood estimate of the win rate is an unbiased estimate because the expected value of the estimate is equal to the true win rate.

[/ QUOTE ]
I'm not a statistician, but Bill claims that this statement is just wrong and that maximum likelihood estimators are not necessarily unbiased. I don't think we need to get into that really. He's at the WSOP now, and it's not the point really.

[bigjohnn]

[ QUOTE ]
The sample mean is always an unbiased estimate of the mean of any distribution by definition.

[/ QUOTE ]

Should this not be qualified by stating that the sample is "random"?

John

[BillChen]

It seems from a lot of the literature, including the Yale stats page and the Association of Physicians that the definition of confidence interval as the interval with the specified value of the parameter in that range. I also asked a couple of my mathematician friends who are doing research outside of statistics who gave a similar definition.

But I think we agree on the point Jerrod was trying to make that a two std deviation radius window from the mean does not mean a 95% probability that the win rate falls within this window, and I don't think this point is well communicated to the world. Without making a value judgement on whether the definition is misleading, I would say most poker players who know about confidence intervals are misled.

Also maximum likelyhood estimators are not always unbiased. A MLE represents the mode or peak of the parameter distribution which may be to the right or to the left of the mean.

I agree it's nice to know what your distribution of observed win rates is given a true win rate--as in most things trying to go backwards from an observed win rate and other information to make a guess at a true win rate is much harder and less well defined, but it's much more important to the poker player. This reminds me of the story of two guys lost in a balloon who ask where they are. "You're in a hot air balloon." "He must have been a mathematician. What he said is precisely correct, but is of no use to us."

Bill Chen

[irchans]

Hi Bill, Jerrod,

Great to see you posting! Do you post anywhere other than 2+2 and rec.gambling.poker?

What is your book about? I really enjoyed Jerrod's posts to RGP on the [0,1] game. Will they be in the book? Is there any place I can preorder the book?

[JohnG]

Jerrod has a journal. Don't know about Bill. I think the rest of your questions are answered somewhere at the following link.

http://www.livejournal.com/users/hgfalling/

[Mason Malmuth]

Hi Bill:

I've only scaned this thread and do not wish to comment on it directly. But I did notice the following:

[ QUOTE ]
Also maximum likelyhood estimators are not always unbiased.

[/ QUOTE ]

It's my understanding that just the opposite is true. That is they are almost always biased. However, as the sample size grow that bias tends to become insignificant, and the bias can also usually be corrected by a factor, and I'm going from memory, of (n)/(n-1), or something like that.

Best wishes,
Mason

[David Sklansky]

All hi falootin math aside, anybody who doesn't integrate all evidence available into their estimates is going to lose bets to those who do.

[BruceZ]

[ QUOTE ]
So first of all, I was wrong about the mathematical definition of confidence intervals. Bill, of course, wasn't, and neither is our book because we aren't specifically addressing the type of confidence interval discussed here.

[/ QUOTE ]

Are you using the term "confidence interval". What does it refer to?


[ QUOTE ]
I apologize for calling the original post "wrong" or "flawed." In the claims that it makes, it is correct. I stand by my statements that it will often be misleading, especially when it is used by the unsuspecting to answer questions for which it is unsuitable. This is not the fault of the original poster, although it is my general feeling that those who answer questions about inferring information about win rates from observed data (the topic we address in our book) in this manner are doing the askers a disservice.

[/ QUOTE ]

Spoken like a true Bayesian. /images/graemlins/laugh.gif Of course any statistical analysis will be misleading if used to answer questions for which it is unsuitable. I believe that it is useful to know that if your win rate were such and such a value, that your results would only be obtained a certain percentage of the time. If this conclusion is stated exactly that way, I don't think anyone should be confused.


[ QUOTE ]
The "probability" definition of a confidence interval is the most commonly known and understood definition; if one types "define: confidence interval" into Google, as I just did, for example, the first result is "a range of values that has a specified probability of containing the rate or trend....(example deleted)."

[/ QUOTE ]

This does not contradict the correct definition of confidence interval I have given, but it is incomplete and highly misleading. Their definition applies only BEFORE the observations are obtained, but not AFTER they are obtained. Before the observations, this range of values can only be given relative to an unknown sample mean, so it is not an absolute range of numbers. For example, before the experiment, we can write:

P(sample mean - 2*sigma < true mean < sample mean + 2*sigma) = 95%.

The range of values in parentheses is the 95% confidence interval. AFTER the experiment, this equation is no longer valid; however, the value in parentheses is still the 95% confidence interval, and this is what google fails to define.

Now if anyone thinks that their definition holds even after the experiment, or if they think that the “range of values” is an absolute set of numbers, then that would be wrong. Absolutely wrong. Confidence and confidence intervals come from statistics, and there is a reason why the term confidence had to be introduced separate from probability. It is not "commonly understood" to mean the same thing as probability. It is commonly misunderstood to mean that. I'm aware that the wrong definition is used all over the place by people who don't understand the underlying statistics. This doesn't make it correct any more than it is correct to say that "2000 volts flowed through someone", though we hear this all the time.


[ QUOTE ]
It is also unhelpful that there is a definition of confidence interval within statistics that is the probability definition that applies to a situation that is not entirely unlike this one to the untrained eye; that is when we know the population mean and want to estimate the outcome of a sample.

[/ QUOTE ]

It's not unhelpful; this IS the definition, and it is the only definition. Yes it is an extremely subtle point that the probability part applies only BEFORE you obtain the outcome of a sample, but not AFTER.


[ QUOTE ]
or if that is not sufficiently compelling, we can poll a hundred smart laypeople and ask them what they think.

[/ QUOTE ]

Ironically, if they all think that their win rate lies in their 95% confidence interval, then on average 95 of them will be right. /images/graemlins/smile.gif


[ QUOTE ]
[ QUOTE ]
It is interesting that you used the word "biased" because this has a precise mathematical definition. Bayesian estimates are biased as a result of taking into account our preconceived notions about the prior probability distribution. On the other hand, the maximum likelihood estimate of the win rate is an unbiased estimate because the expected value of the estimate is equal to the true win rate.

[/ QUOTE ]
I'm not a statistician, but Bill claims that this statement is just wrong and that maximum likelihood estimators are not necessarily unbiased.

[/ QUOTE ]

That's right they aren't, and neither are Bayesian estimators. I explained what I meant by this in my response to Siegmund. The important point is that when we do a maximum likelihood analysis for win rate (as Homer did), we use the sample mean, and this is always unbiased for any distribution.

[BruceZ]

[ QUOTE ]
It seems from a lot of the literature, including the Yale stats page and the Association of Physicians that the definition of confidence interval as the interval with the specified value of the parameter in that range/

[/ QUOTE ]

You'll have to provide links if you want me to debunk these pages. I couldn't find them by searching for those terms. From what you stated, I can't tell if they are defining this correctly or not. It depends on whether they say the parameter has the specified probability of lying in that range computed after the experiment. If they state that, then they are wrong. That's the whole reason the term "confidence" had to be introduced, separate from probability.


[ QUOTE ]
I also asked a couple of my mathematician friends who are doing research outside of statistics who gave a similar definition.

[/ QUOTE ]

It wouldn’t surprise me if they got it wrong. Mathematicians are often not statisticians.


[ QUOTE ]
But I think we agree on the point Jerrod was trying to make that a two std deviation radius window from the mean does not mean a 95% probability that the win rate falls within this window

[/ QUOTE ]

Right, it means there is a 95% confidence that the win rate falls within this window.


[ QUOTE ]
Also maximum likelyhood estimators are not always unbiased.

[/ QUOTE ]

I agree. See my response to Siegmund for what I really meant. We are talking about using the sample mean, which is always an unbiased estimator of the mean for any distribution by definition.


[ QUOTE ]
This reminds me of the story of two guys lost in a balloon who ask where they are. "You're in a hot air balloon." "He must have been a mathematician. What he said is precisely correct, but is of no use to us."

[/ QUOTE ]

I think that maximum likelihood estimation is of considerably more use than that. In fact, it was invented to be a practical alternative to Bayesian estimation that anyone can perform easily, and it is the most widely used method of estimation. Certainly it is useful for a poker player to realize, for example, that if he were really a winning player, that results as bad as his would be obtained less than 5% of the time. That's not just a lot of hot air. /images/graemlins/smile.gif

[BruceZ]

[ QUOTE ]
[ QUOTE ]
Also maximum likelyhood estimators are not always unbiased.

[/ QUOTE ]

It's my understanding that just the opposite is true. That is they are almost always biased.

[/ QUOTE ]


They are often biased, but see my response to Siegmund for what I meant. When we perform a maximum likelihood analysis for the win rate, we use the sample mean which is always an unbiased estimator of the mean of any distribution by definition.


[ QUOTE ]
However, as the sample size grow that bias tends to become insignificant, and the bias can also usually be corrected by a factor, and I'm going from memory, of (n)/(n-1), or something like that.

[/ QUOTE ]

That's for estimating the variance of a normal distribution. For the mean of a normal distribution, the maximum likelihood estimator is the sample mean, and it is unbiased.

[BruceZ]

[ QUOTE ]
All hi falootin math aside, anybody who doesn't integrate all evidence available into their estimates is going to lose bets to those who do.

[/ QUOTE ]

Good, then you agree that we shouldn't ignore the evidence provided by maximum likelihood estimators and confidence intervals. /images/graemlins/smile.gif

No "hi falootin" math here. Just elementary statistics.

[BruceZ]

[ QUOTE ]
[ QUOTE ]
The sample mean is always an unbiased estimate of the mean of any distribution by definition.

[/ QUOTE ]

Should this not be qualified by stating that the sample is "random"?

[/ QUOTE ]

No, because that is already part of the definition of the sample mean. Specifying it again would be redundant.

[bigjohnn]

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
The sample mean is always an unbiased estimate of the mean of any distribution by definition.

[/ QUOTE ]

Should this not be qualified by stating that the sample is "random"?

[/ QUOTE ]

No, because that is already part of the definition of the sample mean. Specifying it again would be redundant.

[/ QUOTE ]

The definition of a sample mean is "the mean of a sample". It makes no claim about the nature of the sample i.e. random or non-random. Surely you need to qualify it.

[BruceZ]

[ QUOTE ]
The definition of a sample mean is "the mean of a sample". It makes no claim about the nature of the sample i.e. random or non-random. Surely you need to qualify it.

[/ QUOTE ]

No, it's the mean of a random sample. I'm using the definition of the sample mean from DeGroot 2nd edition p. 227, which states that the sample mean is the arithmetic average of independent and identically distributed random variables. Also see this page (http://mathworld.wolfram.com/SampleMean.html) which states without qualification that the sample mean is an unbiased estimator of the population mean. This page doesn't even make clear that the samples are random, though any idiot would realize that they have to be. So no, I don't have to qualify it, and if I did I would have.