How does computing the bias of model parameters make sense? I've been studying statistics recently and was thinking about the fact that computing the expectation of a random variable $E(X)$ only really makes sense if $X$ is a random variable defined over a vector space. This is true since for $$E(X) = \int{p(x)\cdot x}$$ to be defined the space needs an addition +, and multiplication by scalar $\cdot$ structure. But in general if you have a family of distributions parametrized by $\theta$, $p(x|\theta)$. $\theta$ does not have a natural vector space structure, and in general only at most has the structure of manifold with possibly a metric given by the fisher information matrix.
So how would it make sense to talk about $E(\theta)$?
Even in the basic case of the estimator for variance of a probability distribution $$\sigma^2 = \frac{\sum_i{(x_i - \mu)^2}}{n-1}$$ I don't see how computing $E(\sigma^2)$ makes sense. the space of variance does not have any natural addition or multiplication by scalar structure.
 A: Note that I use somewhat different notation from you, particularly I distinguish parameters (Greek letters $\mu,\sigma^2$) from the random variables that are used to estimate them ($\bar X, S^2$); I also use capital letter for random variables, as usually done in statistics. I'm making reference to your comment (namely that $S^2=\frac{\sum(X_i-\bar X)^2}{n-1}$ is unbiased for $\sigma^2$, but $S$ is biased for $\sigma$). You are right about this, and you are right about the fact that requiring estimators to be unbiased can be problematic, particularly in cases in which the sampling distribution of the estimator is asymmetric (as for $S^2$). Generally the problem of estimating the variance is not symmetric (regarding over- and underestimation), and it is questionable whether the the mean/expected value of the sampling distribution provides the most useful summary of it.
So I agree with your skepticism. However, I'd not go as far as saying that in such cases studying bias doesn't make sense. We should separate here the question whether
(a) something useful about an estimator can be learnt from it, or (b) whether we demand that an estimator should be unbiased.
For sure I'd say that bias computations are useful; in fact highlighting that estimating $\sigma$ as a problem is in the sense of bias different from estimating $\sigma^2$ is in itself an interesting insight. In an application, one may think about whether relevant deviation is more appropriately be formalised in terms of $\sigma$ or $\sigma^2$, or even $\log(\sigma)$ (which is no longer constrained to be positive, which is a source of asymmetry when estimating a variance).
It is probably often of practical relevance whether an estimator is on average correct. Particularly if there is a statistical decision problem with different parties involved (one of which may hope for a higher parameter value, another for a lower one), unbiasedness is something that can be used to convince the parties that the estimation method used for the decision is fair in the sense of balancing possible high and low deviations from the true value. That said, as I wrote above, it would not be correct to claim that unbiasedness has some "absolute" legitimisation, and that biased estimators should never be considered if unbiased ones exist. In fact, a smaller mean squared error may be achieved (in an asymmetric situation such as variance estimation) with a biased than with an unbiased estimator, and in some applications this can be seen as more important than unbiasedness.
Unbiasedness also makes some theoretical derivations regarding estimators easier and clearer (such as the theory of minimum variance unbiased estimation).
