1
$\begingroup$

It seems to me that bias = variance = 0 but MSE > 0, possibly very high, so clearly my intuition, and math, are wrong.

For a training set $T$ and a regression problem let $M(T) = \text{Ave}(y(T))$. So if p = 99% this is a very "good" model and if p = 50.1% this is a very "bad" model.

As $|T| \rightarrow \infty$ this model becomes very stable (law of large numbers) so it is low variance, in the sense that there is an $m$ so that if $|T| > m$ then the variance will be bound above by $\epsilon$ where $\epsilon < MSE/100$, so that it is not the driver of MSE in the bias-variance decomposition (and in turn the MSE is bounded below by $\sigma^2$ where $\sigma^2$ is irreducible error which is a constant with respect to the model). It also approaches the true mean (also law of large numbers) so it is unbiased. Yet it's clearly a terrible model with high mean squared error unless the target concept itself has very low variance absolutely. What gives?

I should be able to work this out myself from the definition of MSE:

$$MSE(\hat\theta) = \mathbb E[(\hat\theta - \theta)^2]$$ which expands into the bias-variance decomposition

$$\left(\mathbb E(\hat\theta)-\theta\right)^2 + \mathbb E\left[\left(\hat\theta-\mathbb E(\hat\theta)\right)^2\right]$$

The reason I can't work this out myself is I don't understand what the expected values are taken over: the training set $T$ that trains the estimator/model $\hat\theta$ so that $\hat\theta$ is a random variable? The underlying distribution (unknown) $\mathcal D$ over which the data is drawn? (This isn't as bad as it sounds - by LLN the error over a test data set $T'$ is a very good, unbiased estimator for the error over $\mathcal D$, which is pretty much the only reason machine/statistical learning is possible in the first place).

So an explanation of what the $\mathbb E$ are taken over would probably suffice for me to work out the situation with my $\mu$-estimator above. I'm likely misunderstanding the definition of bias or variance or both.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.