The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the best linear unbiased estimator (BLUE) for the coefficients in a linear regression (given some conditions on the errors). I can understand why we want an unbiased and minimum-variance ("best") estimator, but why linear? Why not an estimator that depends on any other power (square, square root, etc) of the data?

More specifically, for an $n\times m$ data matrix $X$ predicting an $n \times 1$ response vector $y$ in the model $y = \beta X + \epsilon$, the OLS estimator for the coefficients $\beta$ is,

$$\hat\beta = (X^TX)^{-1}X^Ty = Cy.$$

Thus each $\hat\beta_j$ can be defined linearly in terms of $y_i$, as

$$\hat\beta_j = c_{j0} y_0 + c_{j1} y_1 + c_{j2} y_2 + \cdots,$$

and is therefore a linear estimator. Is there a particular reason we don't consider non-linear estimator, for example, of the form,

$$ \tilde\beta_j = Cy^a = c_{j0} y_0^a + c_{j1} y_1^a + c_{j2} y_2^a + \cdots $$

  • 4
    $\begingroup$ But we do consider nonlinear estimators, the lasso is a such. $\endgroup$ – kjetil b halvorsen Apr 29 '17 at 16:20
  • 1
    $\begingroup$ Sorry, my question might not be very clear. I wouldn't count lasso, since (afaik) while it is nonlinear, it is neither unbiased nor lower variance than OLS. Maybe I should rephrase: Are there unbiased, non-linear estimators with lower variance than the OLS estimator? $\endgroup$ – user126350 Apr 29 '17 at 21:51

In mathematics rarely things are developed in the way they're presented in textbooks. That's the real reason. Here's the explanation.

First, someone came up with a problem to fit $$y=X\beta+\varepsilon$$, i.e. find the "best" in some respect set of parameters $\beta$. Whoever did this didn't think that the solution would be a liner combination of $y$'s. He simply thought about what would be the criterion to pick the "best" solution, and came up with minimizing the sum of squared errors $\varepsilon'\varepsilon$. This is a very reasonable criterion for many people. So, he went on and formulated the optimization problem: $$\min_\beta \varepsilon'\varepsilon=\min_\beta(y-X\beta)'(y-X\beta)$$

When the guy solved the problem, he was amazed that the solution turned out to be a linear combination of $y$'s: $$(X'X)^{-1}X'y$$

He wasn't looking for solutions that are BLUE or linear. He was just looking for a solution of least squares problem. Then his friends jumped on to study this solution from different angles and came up with Gauss-Markov theorem, BLUE etc.

After this was all done people today look at all kinds of formulations of "best" solution criteria, they're not simply sums of squared errors anymore. Some people want to also have "small" $\beta$, which leads to all kinds of shrinkage methods that are not BLUE or linear anymore, and so on.

I like your question a lot because it separates out the linear model specification $X\beta$ on independent variables and the fact that the solution is a linear combination of dependent variables $Cy$. In order to come from the latter to the former one needs special kind of goodness-of-fit criteria, such as minimum of sum of squares. Other goodness-of-fit criteria may lead to non-linear (on $y$) solutions.

  • $\begingroup$ Thanks for this answer! It makes a lot of sense that the solution to choosing the parameters of a linear model will depend on the goodness of fit criteria. On further consideration, I think my real question is then if this problem could have been solved different: Are there unbiased, non-linear estimators with lower variance than the OLS estimator? $\endgroup$ – user126350 Apr 29 '17 at 21:56
  • $\begingroup$ if it's a different question, then post it separately. that way it'll be easier for those who ask the same question in the future. $\endgroup$ – Aksakal Apr 29 '17 at 22:00
  • $\begingroup$ The OLS solution is simply the best when you have normal errors, among both linear on nonlinear solutions. If you drop the normality then things change, i's not the best anymore, it's only BLUE $\endgroup$ – Aksakal Apr 29 '17 at 22:04
  • $\begingroup$ I have worked with lowest sum-of-squared relative error, rather than lowest sum-of-squared absolute error. In such cases, "lowest variance" means "lowest relative variance". $\endgroup$ – James Phillips Apr 30 '17 at 1:42
  • 1
    $\begingroup$ @Aksakal Can you point to a proof that OLS is best "among both linear or nonlinear solutions" with normal errors? This is exactly what I want to know. $\endgroup$ – user126350 May 10 '17 at 16:28

When the error term is not Gaussian, it will generally be the case that the best estimators (e.g. in terms of MSE) are not linear.

In some cases, all linear estimators may be arbitrarily bad. (It's not always so clear what all the fuss about being BLUE is, when even the best linear estimator may be terrible.)

So for example if the tails of the conditional distribution of the dependent variable are made heavier and heavier, you need give less and less weight to values further away, or the variance of the parameter estimates can be increased beyond any bound.

[Nonlinear estimators include more than just powers, though.]

  • $\begingroup$ I'm interested only in the case of Gaussian errors. Certainly, for non-normal errors OLS is not optimal. $\endgroup$ – user126350 May 10 '17 at 16:31
  • $\begingroup$ You make no mention of $\epsilon$ being Gaussian in the question -- indeed, you seem to have just denied the premise of your own question, then; if the error is jointly Gaussian, and the model is linear then ML will be linear; this follows directly from the quadratic form of the exponent in the Gaussian. $\endgroup$ – Glen_b -Reinstate Monica May 10 '17 at 16:50
  • $\begingroup$ Indeed, I really bungled the original question! Can you elaborate on "the model is linear then ML will be linear; this follows directly from the quadratic form of the exponent in the Gaussian"? What's ML and how does its linearity follow from Gaussian errors? $\endgroup$ – user126350 May 10 '17 at 17:44
  • $\begingroup$ ML=maximum likelihood (sorry). But this is essentially a new question. Consider that in the joint density of errors $\epsilon$, the exponent is $-\frac12\epsilon^\top \Sigma^{-1}\epsilon$ where in ordinary regression $\epsilon=(y-X\beta)$, and $\Sigma=\sigma^2I$. Then $-2\log L_\beta = k+c_\sigma(y-X\beta)^\top(y-X\beta)$, and (handwaving detail in relation to $\sigma^2$) the argmax of $L$ is the minimizer of $(y-X\beta)^\top(y-X\beta) = y^\top y -2\beta^\top X^\top y + \beta^\top X^\top X\beta$. If we take the derivative with respect to $\beta$, setting=0 and solving for $\hat\beta$ ...ctd $\endgroup$ – Glen_b -Reinstate Monica May 10 '17 at 22:44
  • $\begingroup$ ctd ... we get the linear estimator $\hat{\beta}=(X^\top X)^{-1}X^\top y$. Similar results apply for other choices of $\Sigma$ as long as $\epsilon$ is linear in $\beta$ and $y$. e.g. you could generalize to some unspecified $\epsilon=B y - A\beta$ for some conformal matrices $B$ and $A$ (presumably some function of a set of predictors). When you expand $-2\log L$ the term $\epsilon^\top \Sigma^{-1}\epsilon$ has a term in $\beta^\top My$ (or equivalently in $y^\top M^\top \beta$) & a term like $\beta^\top Q_\Sigma \beta$ and so we end up with $\hat\beta=Q_\Sigma^{-1}My$ $\endgroup$ – Glen_b -Reinstate Monica May 10 '17 at 22:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.