Skip to main content
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
Source Link

is a biased estimator of the population standard deviation ($\sigma$). Note, in that formula we decrement the degrees of freedom of $n$ by 1 and dividing by $n-1$, i.e., we do some correction, but it is only asymptotically correct, and $n-3/2$ would be a better rule of thumb. For our $x_2-x_1=d$ example the $\text{SD}$ formula would give us $SD=\frac{d}{\sqrt 2}\approx 0.707d$, a statistically implausible minimum value as $\mu\neq \bar{x}$, where a better expected value ($s$) would be $E(s)=\sqrt{\frac{\pi }{2}}\frac{d}{\sqrt 2}=\frac{\sqrt\pi }{2}d\approx0.886d$. For the usual calculation, for $n<10$, $\text{SD}$s suffer from very significant underestimation called small number bias, which only approaches 1% underestimation of $\sigma$ when $n$ is approximately $25$. Since many biological experiments have $n<25$, this is indeed an issue. For $n=1000$, the error is approximately 25 parts in 100,000. In general, small number bias correctionsmall number bias correction implies that the unbiased estimator of population standard deviation of a normal distribution is

Hint: (But not the answer) see How can I find the standard deviation of the sample standard deviation from a normal distribution?How can I find the standard deviation of the sample standard deviation from a normal distribution?.

is a biased estimator of the population standard deviation ($\sigma$). Note, in that formula we decrement the degrees of freedom of $n$ by 1 and dividing by $n-1$, i.e., we do some correction, but it is only asymptotically correct, and $n-3/2$ would be a better rule of thumb. For our $x_2-x_1=d$ example the $\text{SD}$ formula would give us $SD=\frac{d}{\sqrt 2}\approx 0.707d$, a statistically implausible minimum value as $\mu\neq \bar{x}$, where a better expected value ($s$) would be $E(s)=\sqrt{\frac{\pi }{2}}\frac{d}{\sqrt 2}=\frac{\sqrt\pi }{2}d\approx0.886d$. For the usual calculation, for $n<10$, $\text{SD}$s suffer from very significant underestimation called small number bias, which only approaches 1% underestimation of $\sigma$ when $n$ is approximately $25$. Since many biological experiments have $n<25$, this is indeed an issue. For $n=1000$, the error is approximately 25 parts in 100,000. In general, small number bias correction implies that the unbiased estimator of population standard deviation of a normal distribution is

Hint: (But not the answer) see How can I find the standard deviation of the sample standard deviation from a normal distribution?.

is a biased estimator of the population standard deviation ($\sigma$). Note, in that formula we decrement the degrees of freedom of $n$ by 1 and dividing by $n-1$, i.e., we do some correction, but it is only asymptotically correct, and $n-3/2$ would be a better rule of thumb. For our $x_2-x_1=d$ example the $\text{SD}$ formula would give us $SD=\frac{d}{\sqrt 2}\approx 0.707d$, a statistically implausible minimum value as $\mu\neq \bar{x}$, where a better expected value ($s$) would be $E(s)=\sqrt{\frac{\pi }{2}}\frac{d}{\sqrt 2}=\frac{\sqrt\pi }{2}d\approx0.886d$. For the usual calculation, for $n<10$, $\text{SD}$s suffer from very significant underestimation called small number bias, which only approaches 1% underestimation of $\sigma$ when $n$ is approximately $25$. Since many biological experiments have $n<25$, this is indeed an issue. For $n=1000$, the error is approximately 25 parts in 100,000. In general, small number bias correction implies that the unbiased estimator of population standard deviation of a normal distribution is

Hint: (But not the answer) see How can I find the standard deviation of the sample standard deviation from a normal distribution?.

edited title
Link
amoeba
  • 107.3k
  • 36
  • 321
  • 347

Why we are we using a biased and misleading standard deviation formula for $\sigma$ of a normal distribution?

added 5 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115

Here is ananother example, the minimum number of points in space to establish a linear trend that has an error is three. If we fit these points with ordinary least squares the result for many such fits is a folded normal residual pattern if there is non-linearity and half normal if there is linearity. In the half-normal case our distribution mean requires small number correction. If we try the same trick with 4 or more points, the distribution will not generally be normal related or easy to characterize. Can we use variance to somehow combine those 3-point results? Perhaps, perhaps not. However, it is easier to conceive of problems in terms of distances and vectors. [1]: https://i.sstatic.net/q2BX8.jpg

Here is an example, the minimum number of points in space to establish a linear trend that has an error is three. If we fit these points with ordinary least squares the result for many such fits is a folded normal residual pattern if there is non-linearity and half normal if there is linearity. In the half-normal case our distribution mean requires small number correction. If we try the same trick with 4 or more points, the distribution will not generally be normal related or easy to characterize. Can we use variance to somehow combine those 3-point results? Perhaps, perhaps not. However, it is easier to conceive of problems in terms of distances and vectors. [1]: https://i.sstatic.net/q2BX8.jpg

Here is another example, the minimum number of points in space to establish a linear trend that has an error is three. If we fit these points with ordinary least squares the result for many such fits is a folded normal residual pattern if there is non-linearity and half normal if there is linearity. In the half-normal case our distribution mean requires small number correction. If we try the same trick with 4 or more points, the distribution will not generally be normal related or easy to characterize. Can we use variance to somehow combine those 3-point results? Perhaps, perhaps not. However, it is easier to conceive of problems in terms of distances and vectors. [1]: https://i.sstatic.net/q2BX8.jpg

Notice removed Canonical answer required by Carl
Bounty Ended with Scortchi's answer chosen by Carl
added 2 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 347 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 216 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 53 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 617 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 155 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
Mod Moved Comments To Chat
Tweeted twitter.com/StackStats/status/806791639586869248
added 673 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 4308 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
fix Gamma-ratio in "small-number bias" correction formula
Source Link
GeoMatt22
  • 13.1k
  • 3
  • 39
  • 72
Loading
added 969 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
Notice added Canonical answer required by Carl
Bounty Started worth 50 reputation by Carl
edited title
Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
deleted 68 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 208 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 154 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 154 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading
added 154 characters in body
Source Link
Carl
  • 13.3k
  • 7
  • 55
  • 115
Loading