2 MathJax copy-editing
source | link

It depends on what you are talking about when you say "spread of the data". To me this could mean two things:

  1. The width of a sampling distribution
  2. The accuracy of a given estimate

For point 1) there is no particular reason to use the standard deviation as a measure of spread, except for when you have a normal sampling distribution. The measure $E(|X-\mu|)$ is a more appropriate measure in the case of a Laplace Sampling distribution. My guess is that the standard deviation gets used here because of intuition carried over from point 2). Probably also due to the success of least squares modelling in general, for which the standard deviation is the appropriate measure. Probably also because calculating $E(X^2)$ is generally easier than calculating $E(|X|)$ for most distributions.

Now, for point 2) there is a very good reason for using the variance/standard deviation as the measure of spread, in one particular, but very common case. You can see it in the Laplace approximation to a posterior. With Data $D$ and prior information $I$, write the posterior for a parameter $\theta$ as:

$$p(\theta|DI)=\frac{\exp\left(h(\theta)\right)}{\int \exp\left(h(t)\right)dt}\;\;\;\;\;\;h(\theta)\equiv\log[p(\theta|I)p(D|\theta I)]$$$$p(\theta\mid DI)=\frac{\exp\left(h(\theta)\right)}{\int \exp\left(h(t)\right)\,dt}\;\;\;\;\;\;h(\theta)\equiv\log[p(\theta\mid I)p(D\mid\theta I)]$$

I have used $t$ as a dummy variable to indicate that the denominator does not depend on $\theta$. If the posterior has a single well rounded maximum (i.e. not too close to a "boundary"), we can taylor expand the log probability about its maximum $\theta_{max}$$\theta_\max$. If we take the first two terms of the taylor expansion we get (using prime for differentiation):

$$h(\theta)\approx h(\theta_{max})+(\theta_{max}-\theta)h'(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$$$h(\theta)\approx h(\theta_\max)+(\theta_\max-\theta)h'(\theta_\max)+\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)$$

But we have here that because $\theta_{max}$$\theta_\max$ is a "well rounded" maximum, $h'(\theta_{max})=0$$h'(\theta_\max)=0$, so we have:

$$h(\theta)\approx h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$$$h(\theta)\approx h(\theta_\max)+\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)$$

If we plug in this approximation we get:

$$p(\theta|DI)\approx\frac{\exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$$$p(\theta\mid DI)\approx\frac{\exp\left(h(\theta_\max)+\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)\right)}{\int \exp\left(h(\theta_\max)+\frac{1}{2}(\theta_\max-t)^{2}h''(\theta_\max)\right)\,dt}$$

$$=\frac{\exp\left(\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$$$=\frac{\exp\left(\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)\right)}{\int \exp\left(\frac{1}{2}(\theta_\max-t)^{2}h''(\theta_\max)\right)\,dt}$$

Which, but for notation is a normal distribution, with mean equal to $E(\theta|DI)\approx\theta_{max}$$E(\theta\mid DI)\approx\theta_\max$, and variance equal to

$$V(\theta|DI)\approx \left[-h''(\theta_{max})\right]^{-1}$$$$V(\theta\mid DI)\approx \left[-h''(\theta_\max)\right]^{-1}$$

($-h''(\theta_{max})$$-h''(\theta_\max)$ is always positive because we have a well rounded maximum). So this means that in "regular problems" (which is most of them), the variance is the fundamental quantity which determines the accuracy of estimates for $\theta$. So for estimates based on a large amount of data, the standard deviation makes a lot of sense theoretically - it tells you basically everything you need to know. Essentially the same argument applies (with same conditions required) in multi-dimensional case with $h''(\theta)_{jk}=\frac{\partial h(\theta)}{\partial \theta_j \partial \theta_k}$$h''(\theta)_{jk}=\frac{\partial h(\theta)}{\partial \theta_j \, \partial \theta_k}$ being a Hessian matrix. The diagonal entries are also essentially variances here too.

The frequentist using the method of maximum likelihood will come to essentially the same conclusion because the MLE tends to be a weighted combination of the data, and for large samples the Central Limit Theorem applies and you basically get the same result if we take $p(\theta|I)=1$$p(\theta\mid I)=1$ but with $\theta$ and $\theta_{max}$$\theta_\max$ interchanged: $$p(\theta_{max}|\theta)\approx N\left(\theta,\left[-h''(\theta_{max})\right]^{-1}\right)$$$$p(\theta_\max\mid\theta)\approx N\left(\theta,\left[-h''(\theta_\max)\right]^{-1}\right)$$ (see if you can guess which paradigm I prefer :P ). So either way, in parameter estimation the standard deviation is an important theoretical measure of spread.

It depends on what you are talking about when you say "spread of the data". To me this could mean two things:

  1. The width of a sampling distribution
  2. The accuracy of a given estimate

For point 1) there is no particular reason to use the standard deviation as a measure of spread, except for when you have a normal sampling distribution. The measure $E(|X-\mu|)$ is a more appropriate measure in the case of a Laplace Sampling distribution. My guess is that the standard deviation gets used here because of intuition carried over from point 2). Probably also due to the success of least squares modelling in general, for which the standard deviation is the appropriate measure. Probably also because calculating $E(X^2)$ is generally easier than calculating $E(|X|)$ for most distributions.

Now, for point 2) there is a very good reason for using the variance/standard deviation as the measure of spread, in one particular, but very common case. You can see it in the Laplace approximation to a posterior. With Data $D$ and prior information $I$, write the posterior for a parameter $\theta$ as:

$$p(\theta|DI)=\frac{\exp\left(h(\theta)\right)}{\int \exp\left(h(t)\right)dt}\;\;\;\;\;\;h(\theta)\equiv\log[p(\theta|I)p(D|\theta I)]$$

I have used $t$ as a dummy variable to indicate that the denominator does not depend on $\theta$. If the posterior has a single well rounded maximum (i.e. not too close to a "boundary"), we can taylor expand the log probability about its maximum $\theta_{max}$. If we take the first two terms of the taylor expansion we get (using prime for differentiation):

$$h(\theta)\approx h(\theta_{max})+(\theta_{max}-\theta)h'(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$

But we have here that because $\theta_{max}$ is a "well rounded" maximum, $h'(\theta_{max})=0$, so we have:

$$h(\theta)\approx h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$

If we plug in this approximation we get:

$$p(\theta|DI)\approx\frac{\exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$

$$=\frac{\exp\left(\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$

Which, but for notation is a normal distribution, with mean equal to $E(\theta|DI)\approx\theta_{max}$, and variance equal to

$$V(\theta|DI)\approx \left[-h''(\theta_{max})\right]^{-1}$$

($-h''(\theta_{max})$ is always positive because we have a well rounded maximum). So this means that in "regular problems" (which is most of them), the variance is the fundamental quantity which determines the accuracy of estimates for $\theta$. So for estimates based on a large amount of data, the standard deviation makes a lot of sense theoretically - it tells you basically everything you need to know. Essentially the same argument applies (with same conditions required) in multi-dimensional case with $h''(\theta)_{jk}=\frac{\partial h(\theta)}{\partial \theta_j \partial \theta_k}$ being a Hessian matrix. The diagonal entries are also essentially variances here too.

The frequentist using the method of maximum likelihood will come to essentially the same conclusion because the MLE tends to be a weighted combination of the data, and for large samples the Central Limit Theorem applies and you basically get the same result if we take $p(\theta|I)=1$ but with $\theta$ and $\theta_{max}$ interchanged: $$p(\theta_{max}|\theta)\approx N\left(\theta,\left[-h''(\theta_{max})\right]^{-1}\right)$$ (see if you can guess which paradigm I prefer :P ). So either way, in parameter estimation the standard deviation is an important theoretical measure of spread.

It depends on what you are talking about when you say "spread of the data". To me this could mean two things:

  1. The width of a sampling distribution
  2. The accuracy of a given estimate

For point 1) there is no particular reason to use the standard deviation as a measure of spread, except for when you have a normal sampling distribution. The measure $E(|X-\mu|)$ is a more appropriate measure in the case of a Laplace Sampling distribution. My guess is that the standard deviation gets used here because of intuition carried over from point 2). Probably also due to the success of least squares modelling in general, for which the standard deviation is the appropriate measure. Probably also because calculating $E(X^2)$ is generally easier than calculating $E(|X|)$ for most distributions.

Now, for point 2) there is a very good reason for using the variance/standard deviation as the measure of spread, in one particular, but very common case. You can see it in the Laplace approximation to a posterior. With Data $D$ and prior information $I$, write the posterior for a parameter $\theta$ as:

$$p(\theta\mid DI)=\frac{\exp\left(h(\theta)\right)}{\int \exp\left(h(t)\right)\,dt}\;\;\;\;\;\;h(\theta)\equiv\log[p(\theta\mid I)p(D\mid\theta I)]$$

I have used $t$ as a dummy variable to indicate that the denominator does not depend on $\theta$. If the posterior has a single well rounded maximum (i.e. not too close to a "boundary"), we can taylor expand the log probability about its maximum $\theta_\max$. If we take the first two terms of the taylor expansion we get (using prime for differentiation):

$$h(\theta)\approx h(\theta_\max)+(\theta_\max-\theta)h'(\theta_\max)+\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)$$

But we have here that because $\theta_\max$ is a "well rounded" maximum, $h'(\theta_\max)=0$, so we have:

$$h(\theta)\approx h(\theta_\max)+\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)$$

If we plug in this approximation we get:

$$p(\theta\mid DI)\approx\frac{\exp\left(h(\theta_\max)+\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)\right)}{\int \exp\left(h(\theta_\max)+\frac{1}{2}(\theta_\max-t)^{2}h''(\theta_\max)\right)\,dt}$$

$$=\frac{\exp\left(\frac{1}{2}(\theta_\max-\theta)^{2}h''(\theta_\max)\right)}{\int \exp\left(\frac{1}{2}(\theta_\max-t)^{2}h''(\theta_\max)\right)\,dt}$$

Which, but for notation is a normal distribution, with mean equal to $E(\theta\mid DI)\approx\theta_\max$, and variance equal to

$$V(\theta\mid DI)\approx \left[-h''(\theta_\max)\right]^{-1}$$

($-h''(\theta_\max)$ is always positive because we have a well rounded maximum). So this means that in "regular problems" (which is most of them), the variance is the fundamental quantity which determines the accuracy of estimates for $\theta$. So for estimates based on a large amount of data, the standard deviation makes a lot of sense theoretically - it tells you basically everything you need to know. Essentially the same argument applies (with same conditions required) in multi-dimensional case with $h''(\theta)_{jk}=\frac{\partial h(\theta)}{\partial \theta_j \, \partial \theta_k}$ being a Hessian matrix. The diagonal entries are also essentially variances here too.

The frequentist using the method of maximum likelihood will come to essentially the same conclusion because the MLE tends to be a weighted combination of the data, and for large samples the Central Limit Theorem applies and you basically get the same result if we take $p(\theta\mid I)=1$ but with $\theta$ and $\theta_\max$ interchanged: $$p(\theta_\max\mid\theta)\approx N\left(\theta,\left[-h''(\theta_\max)\right]^{-1}\right)$$ (see if you can guess which paradigm I prefer :P ). So either way, in parameter estimation the standard deviation is an important theoretical measure of spread.

1
source | link

It depends on what you are talking about when you say "spread of the data". To me this could mean two things:

  1. The width of a sampling distribution
  2. The accuracy of a given estimate

For point 1) there is no particular reason to use the standard deviation as a measure of spread, except for when you have a normal sampling distribution. The measure $E(|X-\mu|)$ is a more appropriate measure in the case of a Laplace Sampling distribution. My guess is that the standard deviation gets used here because of intuition carried over from point 2). Probably also due to the success of least squares modelling in general, for which the standard deviation is the appropriate measure. Probably also because calculating $E(X^2)$ is generally easier than calculating $E(|X|)$ for most distributions.

Now, for point 2) there is a very good reason for using the variance/standard deviation as the measure of spread, in one particular, but very common case. You can see it in the Laplace approximation to a posterior. With Data $D$ and prior information $I$, write the posterior for a parameter $\theta$ as:

$$p(\theta|DI)=\frac{\exp\left(h(\theta)\right)}{\int \exp\left(h(t)\right)dt}\;\;\;\;\;\;h(\theta)\equiv\log[p(\theta|I)p(D|\theta I)]$$

I have used $t$ as a dummy variable to indicate that the denominator does not depend on $\theta$. If the posterior has a single well rounded maximum (i.e. not too close to a "boundary"), we can taylor expand the log probability about its maximum $\theta_{max}$. If we take the first two terms of the taylor expansion we get (using prime for differentiation):

$$h(\theta)\approx h(\theta_{max})+(\theta_{max}-\theta)h'(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$

But we have here that because $\theta_{max}$ is a "well rounded" maximum, $h'(\theta_{max})=0$, so we have:

$$h(\theta)\approx h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})$$

If we plug in this approximation we get:

$$p(\theta|DI)\approx\frac{\exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(h(\theta_{max})+\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$

$$=\frac{\exp\left(\frac{1}{2}(\theta_{max}-\theta)^{2}h''(\theta_{max})\right)}{\int \exp\left(\frac{1}{2}(\theta_{max}-t)^{2}h''(\theta_{max})\right)dt}$$

Which, but for notation is a normal distribution, with mean equal to $E(\theta|DI)\approx\theta_{max}$, and variance equal to

$$V(\theta|DI)\approx \left[-h''(\theta_{max})\right]^{-1}$$

($-h''(\theta_{max})$ is always positive because we have a well rounded maximum). So this means that in "regular problems" (which is most of them), the variance is the fundamental quantity which determines the accuracy of estimates for $\theta$. So for estimates based on a large amount of data, the standard deviation makes a lot of sense theoretically - it tells you basically everything you need to know. Essentially the same argument applies (with same conditions required) in multi-dimensional case with $h''(\theta)_{jk}=\frac{\partial h(\theta)}{\partial \theta_j \partial \theta_k}$ being a Hessian matrix. The diagonal entries are also essentially variances here too.

The frequentist using the method of maximum likelihood will come to essentially the same conclusion because the MLE tends to be a weighted combination of the data, and for large samples the Central Limit Theorem applies and you basically get the same result if we take $p(\theta|I)=1$ but with $\theta$ and $\theta_{max}$ interchanged: $$p(\theta_{max}|\theta)\approx N\left(\theta,\left[-h''(\theta_{max})\right]^{-1}\right)$$ (see if you can guess which paradigm I prefer :P ). So either way, in parameter estimation the standard deviation is an important theoretical measure of spread.