1
$\begingroup$

I've been studying statistics recently and was thinking about the fact that computing the expectation of a random variable $E(X)$ only really makes sense if $X$ is a random variable defined over a vector space. This is true since for $$E(X) = \int{p(x)\cdot x}$$ to be defined the space needs an addition +, and multiplication by scalar $\cdot$ structure. But in general if you have a family of distributions parametrized by $\theta$, $p(x|\theta)$. $\theta$ does not have a natural vector space structure, and in general only at most has the structure of manifold with possibly a metric given by the fisher information matrix.

So how would it make sense to talk about $E(\theta)$?

Even in the basic case of the estimator for variance of a probability distribution $$\sigma^2 = \frac{\sum_i{(x_i - \mu)^2}}{n-1}$$ I don't see how computing $E(\sigma^2)$ makes sense. the space of variance does not have any natural addition or multiplication by scalar structure.

Improve this question
$\endgroup$
9
  • $\begingroup$ Are you worried about how to compute the expected value, or is your question rather, given you can compute it, whether it makes any sense, let's say for a practitioner? $\endgroup$
    – Christian Hennig
    Sep 10, 2021 at 14:26
  • 1
    $\begingroup$ I also don't quite understand your problem. $\sigma^2$ is a non-negative real number, you can add and multiply them no problem. Most parameters are actually from $\mathbb{R}^p$. $\endgroup$
    – Christian Hennig
    Sep 10, 2021 at 14:31
  • $\begingroup$ @ChristianHennig, yes I am more worried about it not making sense. As for $\sigma^2$ I am thinking let's say we are trying to fit a normal distribution to some data. You would think to use the "unbiased variance" $\frac{\sum_i{(x_i-\mu)^2}}{n-1}$ but $\sigma^2$ is just an arbitrary choice of a model parameter. it could be $\sigma$ that we use as a model parameter, but $\sqrt{\frac{\sum_i{(x_i-\mu)^2}}{n-1}}$ is a biased estimator for $\sigma$ so now we would need a different value for $\sigma$, and so you would need to change the formula and therefore fit a different gaussian to the data. $\endgroup$
    – puzzleshark
    Sep 10, 2021 at 14:49
  • $\begingroup$ @puzzleshark Can you please fix the typo in your $\LaTeX?$ $\endgroup$
    – Dave
    Sep 10, 2021 at 14:49
  • $\begingroup$ $\{\theta\}$ always has much more than an abstract manifold structure: it also comes with an immersion into $\mathbb{R}^p,$ from which it can inherit its vector operations. But this is scarcely germane, because in the setup you describe nobody would ever consider $E[\theta].$ Those who do write such expressions first must include the information and assumptions that make $\theta$ a random variable. $\endgroup$
    – whuber ♦
    Sep 10, 2021 at 18:10

1 Answer 1

Reset to default
1
$\begingroup$

Note that I use somewhat different notation from you, particularly I distinguish parameters (Greek letters $\mu,\sigma^2$) from the random variables that are used to estimate them ($\bar X, S^2$); I also use capital letter for random variables, as usually done in statistics. I'm making reference to your comment (namely that $S^2=\frac{\sum(X_i-\bar X)^2}{n-1}$ is unbiased for $\sigma^2$, but $S$ is biased for $\sigma$). You are right about this, and you are right about the fact that requiring estimators to be unbiased can be problematic, particularly in cases in which the sampling distribution of the estimator is asymmetric (as for $S^2$). Generally the problem of estimating the variance is not symmetric (regarding over- and underestimation), and it is questionable whether the the mean/expected value of the sampling distribution provides the most useful summary of it.

So I agree with your skepticism. However, I'd not go as far as saying that in such cases studying bias doesn't make sense. We should separate here the question whether (a) something useful about an estimator can be learnt from it, or (b) whether we demand that an estimator should be unbiased.

For sure I'd say that bias computations are useful; in fact highlighting that estimating $\sigma$ as a problem is in the sense of bias different from estimating $\sigma^2$ is in itself an interesting insight. In an application, one may think about whether relevant deviation is more appropriately be formalised in terms of $\sigma$ or $\sigma^2$, or even $\log(\sigma)$ (which is no longer constrained to be positive, which is a source of asymmetry when estimating a variance).

It is probably often of practical relevance whether an estimator is on average correct. Particularly if there is a statistical decision problem with different parties involved (one of which may hope for a higher parameter value, another for a lower one), unbiasedness is something that can be used to convince the parties that the estimation method used for the decision is fair in the sense of balancing possible high and low deviations from the true value. That said, as I wrote above, it would not be correct to claim that unbiasedness has some "absolute" legitimisation, and that biased estimators should never be considered if unbiased ones exist. In fact, a smaller mean squared error may be achieved (in an asymmetric situation such as variance estimation) with a biased than with an unbiased estimator, and in some applications this can be seen as more important than unbiasedness.

Unbiasedness also makes some theoretical derivations regarding estimators easier and clearer (such as the theory of minimum variance unbiased estimation).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the answer, I really appreciate! I can see how it might be useful despite being somewhat questionable. My frustration comes from the fact that we use $S^2=\frac{\sum(X_i-\bar X)^2}{n-1}$ rather than the MLE estimator of $\frac{\sum(X_i-\bar X)^2}{n}$ despite the MLE estimator converging faster (As far as I understand) and being simpler, just because of this “unbiasedness”. The MLE estimator must be unbiased for some parameterization of $\sigma$, $f(\sigma)$ I would assume. (Not sure what $f$ would be). We have just chosen the $\sigma^2$ parameterization arbitrarily. $\endgroup$
    – puzzleshark
    Sep 10, 2021 at 16:17
  • $\begingroup$ My personal opinion on this is that for estimating a variance being "too small" should be penalised more than being "too large" due to asymmetry; in fact $S_{ML}^2=\frac{\sum(X_i-\bar X)^2}{n}$ is better in terms of MSE than $S^2$, but the MSE is also symmetric, and I like $S^2$ more because it avoids errors of the kind of being "too small" a bit more, which I think is good. But that's for sure not the "official reason" why this is mostly used; I rather think one should optimise an asymmetric loss function rather than demanding unbiasedness in this case. $\endgroup$
    – Christian Hennig
    Sep 10, 2021 at 16:33
  • $\begingroup$ $S^2_{ML}$ is occasionally used by the way. It's not that it's forbidden... $\endgroup$
    – Christian Hennig
    Sep 10, 2021 at 16:34
  • $\begingroup$ One thing more: The issue with $S^2$ vs. $S^2_{ML}$ is not the parametrisation, because for $n\to\infty$ both converge to the same $\sigma^2$. $\endgroup$
    – Christian Hennig
    Sep 10, 2021 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.