2
$\begingroup$

An estimator for a set of i.i.d samples drawn from a Gaussian distribution can be

$ \sigma^{2}_{m} = \frac{1}{m}\sum_{i=1}^{m} (x^{(i)} - \mu_{m})^{2}$

this is called the sample variance and $\mu_{m} = \frac{1}{m}\sum_{i=1}^{m} x^{i} $ is the sample mean. As $x^{(i)}$ is a random variable (a sample drawn from the Gaussian distribution), how do we proceed to calculate $\sigma_{m}^{2}$? if $x^{(i)}$ were data points, it would have been obvious but this is not the case here. Should we calculate $\sigma_{i}^{2}$ for $i=1,..,m$ for each sample $x^{(i)}$ and then sum them over?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ A random sample of length $m, m \in \mathbb{N}$ is a set of realizations of $m$ independent, identically distributed (iid) random variables with distribution F: en.wikipedia.org/wiki/Sample_(statistics) Hence, sample consists of observations and not of random variables. Sample variance is calcualted based on the observations as well: en.wikipedia.org/wiki/Variance What exactly does your notation $x^{(i)}$ mean? $\endgroup$
    – Misius
    Jun 17, 2021 at 10:21
  • $\begingroup$ My source is Goodfellow's book (Introduction to deep learning). According to him: ${x^{1}, ... , x^{(m)}}$ are a set of samples that are independently and identically distributed according to a (let's say Gaussian) distribution $p(x^{(i)}) = N(x^{(i)}; \mu, \sigma^{2})$ $\endgroup$
    – samsambakster
    Jun 17, 2021 at 10:52
  • $\begingroup$ I have only access to this book deeplearningbook.org, but there in the notation (page xvi) it is said: $x^{(i)}$ The $i$-th example (input) from a dataset. So I guess they have in mind the observations, or real data points. Do you then maybe have $\mu_n = \frac{1}{m}\sum x^{(i)}$ as well? I would say it is a little uncommon notation from the point of statistical literature, but I guess in the machine learning literature it can be different. $\endgroup$
    – Misius
    Jun 17, 2021 at 11:13
  • $\begingroup$ I am referring to the page 125, Example: Gaussian Distribution Estimator of the Mean. $\endgroup$
    – samsambakster
    Jun 17, 2021 at 11:25
  • 1
    $\begingroup$ To be honest, I do not think that this book is very good in all of these mathematical nuances. I would write $E[X_i]$ or even $E[X]$ since all is iid. Maybe this answer can help you: stats.stackexchange.com/a/396949/324942. In brief, if you are interested in just an estimator for the data that you already have, then $\hat{\sigma}^2$ is just a number calculated from the observations, but if you would like to know the general properties of an estimator (like expectation, bias, asymptotic properties), then you should regard it as a function of random variables. $\endgroup$
    – Misius
    Jun 17, 2021 at 14:49

1 Answer 1

Reset to default
3
$\begingroup$

Edited after reading comments below to avoid making two answers.

Consider $X_1, X_2,...,X_m$ an iid random sample. Let $x_1, x_2,...,x_m$ a realization from $X_1, X_2,...,X_m$. Then $x_1, x_2,...,x_m$ are actual numbers. Since $X_1, X_2,...,X_m$ are iid, we can think of $x_1, x_2,...,x_m$ as m realizations of one stochastic variable $X$ which is iid. with $X_1, X_2,...,X_m$ (or just $X_1$).

$\sigma^2_m=\sigma^2_m(X_1,X_2,...,X_m)$ as described above is called an estimator. The notation with $m$ is because $\sigma^2_m$ depends on the sample size. The stochastic variables $X_1, X_2,...,X_m$ are often suppressed in the notation.

In practice we cannot calculate $\sigma^2_m$ before we have an actual sample $x_1,..,x_m$ of known values. We call $\hat{\sigma^2_m}= \sigma^2_m(x_1,x_2,...,x_m)$ the estimate of $\sigma^2_m$.

For each sample there is one variance estimate, and one mean estimate.

I hope this helps with the confusion.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your answer. So, what you mean is we calculate the estimate for just one random variable? doesn't it violate the definition above. I also need to mention that $\sigma_{m}^{2}$ is actually $\^\sigma_{m}^{2}$ $\endgroup$
    – samsambakster
    Jun 17, 2021 at 11:03
  • $\begingroup$ It is sigma hat. I am not able to write it down correctly. My apologies. $\endgroup$
    – samsambakster
    Jun 17, 2021 at 11:05
  • $\begingroup$ Yes and no. $X_1, X_2,...,X_m$ are each one random variable, however they are i.i.d. so you can think of $x_1,...,x_m$ as $m$ realizations of one random variable, $X$ , that is identical distributed with $X_1$. $\endgroup$
    – Kirsten
    Jun 17, 2021 at 11:58
  • $\begingroup$ Thank you but I don't think I get it. $\endgroup$
    – samsambakster
    Jun 17, 2021 at 13:42
  • $\begingroup$ I tried to edit the answer above to incorporate the comments. I hope it improves the answer. $\endgroup$
    – Kirsten
    Jun 18, 2021 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.