# Other unbiased estimators than the BLUE (OLS solution) for linear models

For a linear model the OLS solution provides the best linear unbiased estimator for the parameters.

Of course we can trade in a bias for lower variance, e.g. ridge regression. But my question is regarding having no bias. Are there any other estimators that are somewhat commonly used, which are unbiased but with a higher variance than the OLS estimated parameters?

If I would have a huge data set I could of course sub-sample it and estimate the parameters with less data, and increase the variance. I assume this could be hypothetically useful.

This is more of a rhetorical question, because when I have read about BLUE estimators, a worse alternative is not provided. I guess that providing worse alternatives could also help people understand the power of BLUE estimators better.

• What about a maximum likelihood estimator? E.g. if you think your data is sampled from a $t$ distribution with a relatively low degrees of freedom parameter ($t(3)$ or $t(4)$ may be characteristic to financial returns), a maximum likelihood estimator would not coincide with OLS but I guess it would still be unbiased. – Richard Hardy May 21 '15 at 17:44
• Relevant: andrewgelman.com/2015/05/11/… – kjetil b halvorsen May 22 '15 at 13:31

One example that comes to mind is some GLS estimator that weights observations differently although that is not necessary when the Gauss-Markov assumptions are met (which the statistician may not know to be the case and hence apply still apply GLS).

Consider the case of a regression of $y_i$, $i=1,\ldots,n$ on a constant for illustration (readily generalizes to general GLS estimators). Here, $\{y_i\}$ is assumed to be a random sample from a population with mean $\mu$ and variance $\sigma^2$.

Then, we know that OLS is just $\hat\beta=\bar y$, the sample mean. To emphasize the point that each observation is weighted with weight $1/n$, write this as $$\hat\beta=\sum_{i=1}^n\frac{1}{n}y_i.$$ It is well-known that $Var(\hat\beta)=\sigma^2/n$.

Now, consider another estimator which can be written as $$\tilde\beta=\sum_{i=1}^nw_iy_i,$$ where the weights are such that $\sum_iw_i=1$. This ensures that the estimator is unbiased, as $$E\left(\sum_{i=1}^nw_iy_i\right)=\sum_{i=1}^nw_iE(y_i)=\sum_{i=1}^nw_i\mu=\mu.$$ Its variance will exceed that of OLS unless $w_i=1/n$ for all $i$ (in which case it will of course reduce to OLS), which can for instance be shown via a Lagrangian:

\begin{align*} L&=V(\tilde\beta)-\lambda\left(\sum_iw_i-1\right)\\ &=\sum_iw_i^2\sigma^2-\lambda\left(\sum_iw_i-1\right), \end{align*} with partial derivatives w.r.t. $w_i$ set to zero being equal to $2\sigma^2w_i-\lambda=0$ for all $i$, and $\partial L/\partial\lambda=0$ equaling $\sum_iw_i-1=0$. Solving the first set of derivatives for $\lambda$ and equating them yields $w_i=w_j$, which implies $w_i=1/n$ minimizes the variance, by the requirement that the weights sum to one.

Here is a graphical illustration from a little simulation, created with the code below:

EDIT: In response to @kjetilbhalvorsen's point I also include the median of the $y_i$ in the plot.

We observe that both estimators are unbiased, but the estimator that uses weights $w_i=(1\pm\epsilon)/n$ as weights for either half of the sample is more variable.

n <- 100
reps <- 1e6

epsilon <- 0.5
w <- c(rep((1+epsilon)/n,n/2),rep((1-epsilon)/n,n/2))

ols <- rep(NA,reps)
weightedestimator <- rep(NA,reps)

for (i in 1:reps)
{
y <- rnorm(n)
ols[i] <- mean(y)
weightedestimator[i] <- crossprod(w,y)

• Another possibility would be (robust) estimators based on minimizing a criterion such as $W=\sum_{i=1}^n w(e_i)$ where $e_i$ is the ith residual and $w$ is some symmetric function, convex or non-convex, with (global) minimum at 0, $w(0)=0$. The Huber estimator would be an example. – kjetil b halvorsen May 22 '15 at 13:23