It seems to me that bias = variance = 0 but MSE > 0, possibly very high, so clearly my intuition, and math, are wrong.

For a training set $T$ and a regression problem let $M(T) = \text{Ave}(y(T))$. So if p = 99% this is a very "good" model and if p = 50.1% this is a very "bad" model.

As $|T| \rightarrow \infty$ this model becomes very stable (law of large numbers) so it is low variance, in the sense that there is an $m$ so that if $|T| > m$ then the variance will be bound above by $\epsilon$ where $\epsilon < MSE/100$, so that it is not the driver of MSE in the bias-variance decomposition (and in turn the MSE is bounded below by $\sigma^2$ where $\sigma^2$ is irreducible error which is a constant with respect to the model). It also approaches the true mean (also law of large numbers) so it is unbiased. Yet it's clearly a terrible model with high mean squared error unless the target concept itself has very low variance absolutely. What gives?

I should be able to work this out myself from the definition of MSE:

$$MSE(\hat\theta) = \mathbb E[(\hat\theta - \theta)^2]$$ which expands into the bias-variance decomposition

$$\left(\mathbb E(\hat\theta)-\theta\right)^2 + \mathbb E\left[\left(\hat\theta-\mathbb E(\hat\theta)\right)^2\right]$$

The reason I can't work this out myself is I don't understand what the expected values are taken over: the training set $T$ that trains the estimator/model $\hat\theta$ so that $\hat\theta$ is a random variable? The underlying distribution (unknown) $\mathcal D$ over which the data is drawn? (This isn't as bad as it sounds - by LLN the error over a test data set $T'$ is a very good, unbiased estimator for the error over $\mathcal D$, which is pretty much the only reason machine/statistical learning is possible in the first place).

So an explanation of what the $\mathbb E$ are taken over would probably suffice for me to work out the situation with my $\mu$-estimator above. I'm likely misunderstanding the definition of bias or variance or both.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.