22
$\begingroup$

When we calculate mean and variance using the two equations taught in school:

  1. $\mu = \frac{1}{N}\sum_{i=1}^N{x_i}$
  2. $\sigma^2 = \frac{1}{N}\sum_{i=1}^N{(x_i-\mu)^2}$

Then do we assume, that the data are normally distributed? Since the equations come from maximum likelihood of normal distribution estimation and to my knowledge, they should.

$\endgroup$
6
  • 11
    $\begingroup$ I cannot see any reason why descriptive statistics should be anything to do with inferential statistics. Can you edit to expand on why you think normality is necessary to calculate the mean and variance/ $\endgroup$
    – mdewey
    Commented Oct 25, 2020 at 15:15
  • 12
    $\begingroup$ I don't understand the downvote. The answer is that we do not have to assume normality, sure, but the question is legitimate. $\endgroup$
    – Dave
    Commented Oct 25, 2020 at 16:28
  • 3
    $\begingroup$ When you calculate the sample mean, you in some sense assume each observation is equally likely because you weight by the factor 1/N $\endgroup$ Commented Oct 25, 2020 at 16:59
  • 1
    $\begingroup$ Related: math.stackexchange.com/questions/3331917/… $\endgroup$ Commented Oct 26, 2020 at 22:12
  • 1
    $\begingroup$ if we do not have to assume normality, why is the question legitimate? There must be a reason why the sample moments, and their derivation from maximum likelihood estimation, make people automatically think about the Gaussian. What explains this behavior of association $\endgroup$
    – develarist
    Commented Oct 27, 2020 at 15:55

6 Answers 6

25
$\begingroup$

No, those equations come directly from the mean and variance formulae in terms of expected value, considering the collected data as a population.

$$\mu = \mathbb{E}\big[X\big]$$

$$\sigma^2 = \mathbb{E}\big[\big(X-\mu\big)^2\big]$$

Since you have a finite number of observations, the distribution is discrete,$^{\dagger}$ and the expected value is a sum.

$$\mu = \mathbb{E}\big[X\big] = \sum_{i=1}^N p(x_i)x_i = \sum_{i=1}^N \dfrac{1}{N}x_i = \dfrac{1}{N}\sum_{i=1}^Nx_i$$

$$\sigma^2 = \mathbb{E}\big[\big(X-\mu\big)^2\big] = \sum_{i=1}^N p(x_i)(x_i - \mu)^2 = \sum_{i=1}^N \dfrac{1}{N}(x_i - \mu)^2 = \dfrac{1}{N}\sum_{i=1}^N (x_i - \mu)^2$$

(To get from $p(x_i)$ to $\dfrac{1}{N}$, note that each individual $x_i$ has probability $1/N$.)

This is why the $\dfrac{1}{N}\sum_{i=1}^N (x_i - \mu)^2$ gets called the "population" variance. It literally is the population variance if you consider the observed data to be the population.

$^{\dagger}$ This is a sufficient, but not necessary, condition for a discrete distribution. A Poisson distribution is an example of a discrete distribution with infinitely many values.

$\endgroup$
7
  • 2
    $\begingroup$ I thought my answer was too long (and maybe it is), but you have mentioned a few important things I left out. (+1). $\endgroup$
    – BruceET
    Commented Oct 25, 2020 at 16:52
  • $\begingroup$ is it more correct to say mean and variance come directly from expected values, or mean and variance come from maximum likelihood estimation? $\endgroup$
    – develarist
    Commented Oct 27, 2020 at 15:58
  • 1
    $\begingroup$ @develarist There is no estimation going on. $\endgroup$
    – Dave
    Commented Oct 27, 2020 at 16:07
  • $\begingroup$ Why does the question say "the equations come from maximum likelihood of normal distribution estimation". is she wrong? $\endgroup$
    – develarist
    Commented Oct 27, 2020 at 16:09
  • $\begingroup$ Those happen to be maximum likelihood estimates, but if you’re considering the observations alone, you would use the e population equations. It is defendable to use, for instance, median as an estimator of mean. Saying that the mean of $\{1,2,9\}$ is 2 cannot be defended. $\endgroup$
    – Dave
    Commented Oct 27, 2020 at 16:28
11
$\begingroup$

You are mixing descriptive statistics of a sample (such as $\bar X, S)$ with parameters of a population (such as $\mu, \sigma),$ and description of a sample with estimation of parameters.

Describing sample center or location.

The correct version of the sample mean of a sample $X_i, X_2, \dots X_n$ of size $n$ is $\bar X = \frac 1 n \sum_{i=1}^n X_i.$ Many authors reserve $N$ for the size of the population. The sample mean $\bar X$ is a descriptive statistic. It is one way to describe the "center" of a sample.

Some alternative ways to describe the center or location of a sample are (a) the sample median, which is the middle value when data are sorted from smallest to largest (or halfway between the middle two values if the sample size is even), (b) the midrange, which is halfway between the largest and smallest sample values, and (c) the mode which is the value that occurs most often in the sample (if there is one such value).

If you have a sample of seven test scores (78, 96, 84, 92, 88, 75, 51), then R statistical software gives the following summary of the data:

 x = c(78, 96, 84, 92, 88, 75, 51)
 summary(x)
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   51.00   76.50   84.00   80.57   90.00   96.00
 length(x);  sum(x);  sum(x)/length(x)
 [1] 7          # sample size
 [1] 564        # total of seven observations
 [1] 80.57143   # mean (to more places than above)
 sort(x) 
 [1] 51 75 78 84 88 92 96
 min(x);  max(x);  median(x)
 [1] 51        # smallest
 [1] 96        # largest
 [1] 84        # middle value of 7 sorted values.

The midrange (96 + 51)/2 = 73.5 is not given by summary; this sample has no mode.

For small samples an effective graphical description may be the stripchart (or dotplot):

stripchart(x, pch=19)

enter image description here

For larger samples a boxplot or histogram (not shown here) may be used.

The choice whether to use sample mean, sample median, sample midrange (or some other descriptive statistic) depends on the nature of the data and on one's purpose in finding the center or location of the sample.

By contrast, $\mu$ denotes the population mean $\mu.$ So if you have a finite population of size $N$ with elements $X_i,$ then your equation (1) would be the definition of the population mean $\mu.$ [For a theoretical infinite infinite population specified in terms of its density function $f(x),$ the population mean is defined as $\mu = \int xf(x)\,dx,$ where the integral is taken over the interval of all possible population values, provided that the integral exists. (For many of the distributions used in statistical work the population mean $\mu$ exists; Student's t distribution with one degree of freedom is a well-known exception.)]

Describing sample variation and spread.

The usual definition of the sample variance is $S^2=\frac{1}{n-1}\sum_{i-1}^n (X_i - \bar X)^2.$ [In a few textbooks the denominator $n$ is used.] The units of the sample variance are the square of the units of the sample. [So if the sample is heights of students in inches, then the units of the sample variance are square inches.] The sample variance describes the variation of a sample, A related descriptive statistic for sample variation is the sample standard deviation $S = \sqrt{\frac{1}{n-1}\sum_{i-1}^n (X_i - \bar X)^2}.$ its units are the same as the units of the sample.

Some alternative ways to describe the variation of a sample are the sample range (largest sample value minus smallest) and the midrange, which is the range of the middle half of the data (upper quartile minus lower quartile). [There are still other descriptions of sample variation; some are based on medians.]

For the sample of seven test scores above, the variance and standard deviation are as follows:

var(x);  sd(x)
[1] 224.619
[1] 14.9873

From the summary above, the range is (96 - 51) = 45, and the interquartile range (IQR) is $(90 - 76.4) = 13.6.$

diff(range(x));  IQR(x)
[1] 45
[1] 13.5

(A peculiarity of R is that range returns min and max, so we get the usual sample range by subtraction.)

Estimation of parameters.

Depending on the shape of a population distribution, it may be appropriate to estimate the population mean $\mu$ by the sample mean $\bar X,$ or to estimate the population median $\eta$ (half of the probability on either side) by the sample median. Also, it may be appropriate to estimate the population variance $\sigma^2$ by the sample variance $S^2,$ or to estimate the population standard deviation by $\sigma$ by $S.$

Among many, a couple of criteria for a desirable estimator is that it unbiased and that it have the smallest possible variance. Roughly speaking, this amounts to ensuring that on average the estimator is aimed at the right target (unbiasedness) and that the aim is optimally precise (small variance).

This is not the place for a detailed discussion of estimation. However, it is worth mentioning that, for normal data, $S^2$ as defined above is an unbiased estimator for $\sigma^2,$ while the maximum likelihood estimator $\widehat{\sigma^2} = \frac 1 n\sum_{i=1}^n(X_i-\bar X)^2$, with denominator $n,$ has a downward bias, systematically underestimating $\sigma^2.$ Therefore many (but not all) statistics tests use $S^2$ (denominator $n-1)$ as the estimator of $\sigma^2.$ (Perhaps see this related Q&A.)

[As @Dave (+1) makes clear in his Answer, your equation (2), with $N$ in the denominator is the the formula for $\sigma^2$ of a finite population consisting of $N$ possible values, for which the population mean $\mu$ is known.]

$\endgroup$
6
  • $\begingroup$ What do you mean by $i-1$ as summation start? Probably $i=1$. $\endgroup$
    – hans
    Commented Oct 26, 2020 at 11:39
  • 1
    $\begingroup$ Of course, typo fixed. Thanks. $\endgroup$
    – BruceET
    Commented Oct 26, 2020 at 16:42
  • 1
    $\begingroup$ (+1) Worth noting, perhaps, that your $S^2$ & $\bar X$ are unbiased estimators of their population analogues in all families of distributions where those exist. Moreover, in families of distributions for which the order statistic is complete & minimal sufficient, they, also being functions of the order statistic (i.e. symmetric functions), are, by the Lehmann-Scheffé theorem, UMVUE. IIRC, such families include e.g. that of all continuous distributions having means & variances. $\endgroup$
    – Scortchi
    Commented Oct 26, 2020 at 23:41
  • 1
    $\begingroup$ One quibble: you could just as well use a biased estimate of variance as an unbiased one in the denominator of some test statistic, as long as you make a compensating adjustment in your table of critical values. Multiplying a test statistic by a constant factor doesn't change how it orders & partitions the sample space - see e.g. Should the standard deviation be corrected in a Student's T test?. $\endgroup$
    – Scortchi
    Commented Oct 27, 2020 at 1:29
  • $\begingroup$ @Scortchi-ReinstateMonica. Agreed. My link says that using the MLE would require traditional methods of inference for variances using the chi-squared distribution to be altered. Also that many statisticians believe underestimating $\sigma^2$ may be misleading and is a strong argument against the MLE. // Maybe should have repeated that here. // OP is clearly just getting started on this and I hesitated to get deeper than absolutely necessary into the theory. $\endgroup$
    – BruceET
    Commented Oct 27, 2020 at 4:12
4
$\begingroup$

Normality is an ideal case for the ordinary sample mean and variance (as well as other common statistics like least squares regression estimates), but it is certainly not a requirement. After all, normality is never true in practice for any any real data-generating process, nor is it ever true for actual data. So if normality were an absolute requirement, we would never ever ever use the sample mean, and variance, and many other common statistics.

There are things you can say about the usual sample mean and variance under non-normality; for example, Chebychev's inequality tells you that at least $75\%$ of your $x_i$ will be within your $\mu \pm 2\sigma$ range (assuming your equation (2) is called $\sigma^2$), at least $88.9\%$ will be within the $\mu \pm 3\sigma$ range, and in general, at least $100(1 - 1/k^2)\%$ will be within the $\mu \pm k\sigma$ range. These facts do not depend on the source of the $x_i$ data; in fact, the data need not come from any probability model whatsoever.

Additionally, the Central Limit Theorem applies to the sample mean when the data come from a non-normal distribution; this allows you to use the usual normality-assuming confidence interval formula, which involves your "$\mu$" and "$\sigma$," to construct a valid large-sample confidence interval for the mean of the data-generating process, even when that process is non-normal (as long as its variance is finite).

$\endgroup$
4
$\begingroup$

$\mu$ is, indeed, the value that maximizes the Gaussian likelihood, but you have to assume Gaussianity first. You could assume other distributions instead.

Assume you are flipping coins, and you want to estimate the probability $p$ of heads. You toss it $n$ times, and measure $\hat p \times n$ tails. This process assumes a Bernoulli distribution, which likelihood can be stated as:

$$\mathcal L_{\mathcal B} \propto p^{\hat p \times n}(1-p)^{(1-\hat p) \times n}$$

The maximum likelihood estimator for $p$ is $\hat p = (1/n)\sum_i^n t_i = \mathbb E(T)$, where $t_i$ is the result of each toss (1 for heads, 0 for tails).

Here, we are using the expected value to maximize a Bernoulli likelihood.


Distributions have quantities called moments attached to them.

The $n$-th order moment about a value $c$ is defined as, given a probability density described by $f$:

$$\mu_n(c) = \int_{-\infty}^{+\infty} (x-c)^n f(x) dx$$

The mean is the first raw moment (moment about the origin) while the variance is the second central moment (moment about the mean).

\begin{cases}\mathbb E(X) = \mu=\mu_1(\mathbf 0)\\\operatorname{Var}(X)=\mu_2(\mu)=\mu_ 2(\mathbf 0)-\mu^2=\mathbb E(X^2)-\mathbb E(X)^2\end{cases}

These are important quantities defined for any distribution described by $f$, irrespective of if it's Gaussian or not.

$\endgroup$
1
$\begingroup$

One thing that's only been touched on in the answers so far is the (widespread) use of sample mean & variance to estimate their population analogues without making the assumption that the data are from a distribution in any particular parametric family.

For independently, identically distributed observations from distributions in some big non-parametric families (e.g. the family of all continuous distributions having a mean & variance), the order statistic, i.e. the observations put in order from lowest to highest, is complete & minimally sufficient—a rough & ready way of putting this is that reducing the data thus keeps all the information about which distribution they come from while squeezing out all the noise. The sample mean & variance are functions of the order statistic—they're permutation-invariant, they remain the same however you order the observations— & also unbiased: therefore by the Lehmann–Scheffé Theorem they're uniformly minimum-variance unbiased (UMVUE) estimators. They enjoy this property in small samples as much as in large samples where the CLT may justify the approximation of their distribution by a Gaussian (for the construction of tests & confidence intervals).


† Lehmann & Casella (1998), Theory of Point Estimation 2nd edn, Ch.3 "Unbiasedness", §4 "Nonparametric families"

$\endgroup$
1
$\begingroup$

Method of moments

The expressions on the right are sample moments and on the left are parameters of a distribution (in terms of moments of the distribution).

$$\begin{array}{ccl} \overbrace{\mu}^{\substack{\text{parameters of}\\\text{population distribution}\\\text{in terms of moments}}} &=& \overbrace{\frac{1}{N}\sum_{i=1}^N{x_i}}^{\text{sample moments}}\\ \sigma^2 &=& \frac{1}{N}\sum_{i=1}^N{(x_i-\mu)^2} \end{array}$$

Whenever you are setting these two equal then you are employing the method of moments.

You can use this method also when you are not dealing with a normal distribution.

Example: betabinomial distribution

Say we have a population that follows a betabinomial distribution with a fixed size parameter $n$ and unknown parameters $\alpha$ and $\beta$. For this case we can also parameterize the distribution in terms of the mean and variance

$$\begin{array}{rcl} \frac{n \alpha}{\alpha + \beta} &=& \mu\\ \frac{n\alpha\beta(n+\alpha+\beta)}{(\alpha +\beta)^2(\alpha+\beta+1)} &=& \sigma^2 \end{array}$$

and set it equal to the sample moments

$$\begin{array}{rcccccl} \frac{n \hat\alpha}{\hat\alpha + \hat\beta}&=& \hat{\mu} &=& \bar{x} &=&\frac{1}{N}\sum_{i=1}^N{x_i}\\ \frac{n\hat\alpha\hat\beta(n+\hat\alpha+\hat\beta)}{(\hat\alpha +\hat\beta)^2(\hat\alpha+\hat\beta+1)}&=& \hat{\sigma}^2 &=& s^2 &=&\frac{1}{N}\sum_{i=1}^N{(x_i-\bar{x})^2} \end{array}$$

From which estimates for the distribution follow

$$\begin{array}{rcl} \hat\alpha &=& \frac{ n\hat{x}-s^2-\hat{x}^2 }{n ( \frac {s^2}{\hat{x}}-1 ) +\hat{x}} \\ \hat\beta &=&\frac{( n-\hat{x} ) ( n-{\frac {s^2+\hat{x}^2}{\hat{x}}} )}{n ( \frac {s^2}{\hat{x}}-1 ) +\hat{x}} \end{array}$$

With the above estimates $\hat{\alpha}$ and $\hat{\beta}$ the estimated population has the same mean and variance as the sample.

Note

In the case of estimating the parameters of a normal distribution, then the method of moments coincides with the maximum likelihood method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.