confusion regarding mean squared error

Question

I am little confused regarding the concept of MEAN SQUARED ERROR :

my text book says that mean squared error of an estimator say T for estimating the parameter ,say $\theta$ is given by : $ \ $ MSE = $E(T-\theta)^2$ ...... 1 ,

and then they say that estimators with minimum MSE rarely exists in general ,therefore we refer to the class of unbiased estimators and within them we search for the one with minimum MSE :

my question is : the above statement makes me think that is it possible that a biased estimator may have lesser MSE than an unbiased one , but through my previous knowledge i know that $ \ $ $ E(T-A)^2 $ is minimum when $ A = E(T)$ , now for an unbiased estimator $E(T)=\theta $ so that MSE should be minimum for an unbiased estimators only ? please correct me where i am wrong .

Answer 1

Stein's paradox is exactly what you said: there are cases where a biased estimator will have a lower MSE than unbiased estimators. I didn't exactly follow your logic as to why this doesn't make sense. I think the confusion is that you fix $T$ and solve for $A$ but that's should be the other way around - we have $\theta$ and we want to find an estimator $T$ that minimizes the MSE. (or perhaps I didn't understand your claim...)

Answer 2

As Daniel says, I think you are mixing up what is $ T $ and what is $ A $, what is the Random Variable, what is the estimator etc.

Let me tell you my understanding of estimating a Random Variable $VS$ estimating a parameters (frequentist point of view). In both cases I am going to consider that my objective is to minimize the $MSE$:

Let be $X$ a random Variable. If I would want to predict X with a constant minimizing the $MSE$:

$$ x_{mse} = arg\ min _{c} \ E[(X-c)^2] => x_{mse} = E[X] => MSE_{min}= Var(X) $$

Of course when you are working with data, most of the times you don't know the parameters of the R.V $ X $, in this case for example you don't know the mean so you would have to estimate it. This is to estimate a parameter, not a R.V (there is no randomness in the parameter as the frequentist point of view claims) and let me call the parameter we are intesrested in as $\theta$. To estimate a parameter you use data that comes from a $R.V$, perform some operation $T$ on the data $\{X\}_i^n$ such that $ T = T(\{X\}_i^n) $. Here the Random Variable is $ T$ not $\theta$. With this clear now the $MSE$ when estimating the parameter $\theta$ is:

$$ MSE = E[(\theta - T)^2] = Var(\theta-T) + E^2[\theta - T] = Var(T) + bias^2(T) $$

Here I have used the fact that $Var(\theta) = 0$ and $bias(T) = E[\theta - T] = \theta - E[T]$, remember that there is no randomness in $\theta.$ You could come up with more thatn one estimator to estimate $ \theta $, let's say $ T_1 , T_2$. Also let's assume that:

$$ E[T_1] = \theta_1 = \theta => bias(T_1) = = 0 \\ E[T_2] = \theta_2 \neq \theta => bias(T_2) \neq 0 $$

This means that $T_1$ is unbiassed ant $T_2$ is not. So, their $MSE's$ are:

$$ MSE_{T_1} = Var(T_1) + bias^2(T_1) = Var(T_1) \\ MSE_{T_2} = Var(T_2) + bias^2(T_2) $$

Despite that $T_1$ is unbiassed it could have much more greater variance than $T_2$ so that $ Var(T_1) > Var(T_2) + bias^2(T_2)$ and thus beeing an poorer estimator regarding the $MSE$ performance.

So, to wrap up, if you want to estimate a R.V with a constant, the minimum $MSE$ is reached with the constant equal the expected value of the variable but when you are estimating a parameteer (a constant) with data (samples of a Random Variable) is different, there is no "Expected value" of the parameter, only the parameter value itself. Also, normally you use unbiassed estimators but this doesn't implies that the $MSE$ you obtain is the minimum possible because it could have a greater variance.