Estimating Standard Deviation

[irchans]

Does anyone know an unbiased estimator for standard deviation given n samples? I guess that the most commonly used estimator is

<font class="small">Code:</font><hr /><pre>
Sqrt[
Total[(list-Mean[list])^2]/(Length[list]-1)
].
</pre><hr />

But that one is biased. (If you remove the square root, you get an unbiased estimator for variance.)

[doormat]

I don't know a better formula but I found this reference which may or may not be helpful:

Sample standard deviation: The sample variance is an unbiased estimator of the parametric variance. Taking the square root of the sample variance does not give an unbiased estimator of the parametric standard deviation. A rather complicated exact correction factor, and a simpler approximate correction factor, are given on p. 53 of Sokal and Rohlf.

doormat

[BruceZ]

The sample variance is an unbiased estimator of the parametric variance.

Only if you define sample variance as the estimator that irchans has given above which divides by N-1. Many books define the sample variance as the one that divides by N. That is a biased estimator, and this is a constant source of confusion.

I found an online "math encyclopedia" reference that erroneously states that the square root of the unbiased estimator of the variance is an unbiased estimator of the the standard deviation.

Idiots! (http://www.itu.dk/bibliotek/encyclopedia/math/s/s670.htm)

[irchans]

Doormat,

That is exactly what I was looking for. You said page 53 of Sokal and Rohlf. Did you mean the book Biometry: The Principles and Practice of Statistics in Biological Research (http://www.amazon.com/exec/obidos/tg/detail/-/0716724111/qid=1064675074/sr=1-1/ref=sr_1_1/102-9298877-8580912?v=glance&amp;s=books) ?

[irchans]

Bruce,

I must admit that I always used the N-1 formula without even thinking about it when I was trying to estimate standard deviation. I was quite surprised last week when the bias in that formula caused a problem with my work last week.

[irchans]

Is it the curvature of the Sqrt function that causes the unbiased estimator for Variance to be a biased estimator for Standard Deviation?

[BruceZ]

Think of doing N estimates. The average of square roots is not the square root of the average.

It will be unbiased in the limit, so I wouldn't think it would normally cause a problem.

I never thought about it until you brought it up either. /images/graemlins/grin.gif

[doormat]

Biometry, 3rd edition, by R.R. Sokal and F.J. Rohlf, published by W.H. Freeman and Co.

That is how it was cited. There was a Biometry, 2nd edition that is out of print and the book you mention is 3rd edition as listed on Amazon so hopefully the one you mention is the correct one. Sorry but I only found a brief sketchy reference to the book so I can't be sure.

you might check here also:
http://meso.spawar.navy.mil/Docs/MESO-00-A003-6.pdf

[BruceZ]

I sent these guys an email about this, and I got an automatic response from mathworld.wolfram.com. What's interesting is that the mathworld website does not make this error. It states specifically that this is NOT an unbiased estimator. The error only appears in the CRC version which is available on CD, though this was originally based on the mathworld material, so mathworld may have had the error at one time.

The mathworld website is very valuable, and its creator is certainly not an "idiot", though even his site does have quite a few typos. The CRC is an example of a big company exploiting the work of a kid who put a lot of time and effort into compiling lots of valuable information, as you can read about at www.mathworld.wolfram.com. (http://www.mathworld.wolfram.com.) The mathworld website should be used instead, as this is being constantly updated.