# Intuition behind the Formula for an estimator

An estimator for a set of i.i.d samples drawn from a Gaussian distribution can be

$$\sigma^{2}_{m} = \frac{1}{m}\sum_{i=1}^{m} (x^{(i)} - \mu_{m})^{2}$$

this is called the sample variance and $$\mu_{m} = \frac{1}{m}\sum_{i=1}^{m} x^{i}$$ is the sample mean. As $$x^{(i)}$$ is a random variable (a sample drawn from the Gaussian distribution), how do we proceed to calculate $$\sigma_{m}^{2}$$? if $$x^{(i)}$$ were data points, it would have been obvious but this is not the case here. Should we calculate $$\sigma_{i}^{2}$$ for $$i=1,..,m$$ for each sample $$x^{(i)}$$ and then sum them over?

• A random sample of length $m, m \in \mathbb{N}$ is a set of realizations of $m$ independent, identically distributed (iid) random variables with distribution F: en.wikipedia.org/wiki/Sample_(statistics) Hence, sample consists of observations and not of random variables. Sample variance is calcualted based on the observations as well: en.wikipedia.org/wiki/Variance What exactly does your notation $x^{(i)}$ mean? Jun 17, 2021 at 10:21
• My source is Goodfellow's book (Introduction to deep learning). According to him: ${x^{1}, ... , x^{(m)}}$ are a set of samples that are independently and identically distributed according to a (let's say Gaussian) distribution $p(x^{(i)}) = N(x^{(i)}; \mu, \sigma^{2})$ Jun 17, 2021 at 10:52
• I have only access to this book deeplearningbook.org, but there in the notation (page xvi) it is said: $x^{(i)}$ The $i$-th example (input) from a dataset. So I guess they have in mind the observations, or real data points. Do you then maybe have $\mu_n = \frac{1}{m}\sum x^{(i)}$ as well? I would say it is a little uncommon notation from the point of statistical literature, but I guess in the machine learning literature it can be different. Jun 17, 2021 at 11:13
• I am referring to the page 125, Example: Gaussian Distribution Estimator of the Mean. Jun 17, 2021 at 11:25
• To be honest, I do not think that this book is very good in all of these mathematical nuances. I would write $E[X_i]$ or even $E[X]$ since all is iid. Maybe this answer can help you: stats.stackexchange.com/a/396949/324942. In brief, if you are interested in just an estimator for the data that you already have, then $\hat{\sigma}^2$ is just a number calculated from the observations, but if you would like to know the general properties of an estimator (like expectation, bias, asymptotic properties), then you should regard it as a function of random variables. Jun 17, 2021 at 14:49

Consider $$X_1, X_2,...,X_m$$ an iid random sample. Let $$x_1, x_2,...,x_m$$ a realization from $$X_1, X_2,...,X_m$$. Then $$x_1, x_2,...,x_m$$ are actual numbers. Since $$X_1, X_2,...,X_m$$ are iid, we can think of $$x_1, x_2,...,x_m$$ as m realizations of one stochastic variable $$X$$ which is iid. with $$X_1, X_2,...,X_m$$ (or just $$X_1$$).

$$\sigma^2_m=\sigma^2_m(X_1,X_2,...,X_m)$$ as described above is called an estimator. The notation with $$m$$ is because $$\sigma^2_m$$ depends on the sample size. The stochastic variables $$X_1, X_2,...,X_m$$ are often suppressed in the notation.

In practice we cannot calculate $$\sigma^2_m$$ before we have an actual sample $$x_1,..,x_m$$ of known values. We call $$\hat{\sigma^2_m}= \sigma^2_m(x_1,x_2,...,x_m)$$ the estimate of $$\sigma^2_m$$.

For each sample there is one variance estimate, and one mean estimate.

I hope this helps with the confusion.

• Thank you for your answer. So, what you mean is we calculate the estimate for just one random variable? doesn't it violate the definition above. I also need to mention that $\sigma_{m}^{2}$ is actually $\^\sigma_{m}^{2}$ Jun 17, 2021 at 11:03
• It is sigma hat. I am not able to write it down correctly. My apologies. Jun 17, 2021 at 11:05
• Yes and no. $X_1, X_2,...,X_m$ are each one random variable, however they are i.i.d. so you can think of $x_1,...,x_m$ as $m$ realizations of one random variable, $X$ , that is identical distributed with $X_1$. Jun 17, 2021 at 11:58
• Thank you but I don't think I get it. Jun 17, 2021 at 13:42
• I tried to edit the answer above to incorporate the comments. I hope it improves the answer. Jun 18, 2021 at 8:17