8
$\begingroup$

The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the best linear unbiased estimator (BLUE) for the coefficients in a linear regression (given some conditions on the errors). I can understand why we want an unbiased and minimum-variance ("best") estimator, but why linear? Why not an estimator that depends on any other power (square, square root, etc) of the data?

More specifically, for an $n\times m$ data matrix $X$ predicting an $n \times 1$ response vector $y$ in the model $y = \beta X + \epsilon$, the OLS estimator for the coefficients $\beta$ is,

$$\hat\beta = (X^TX)^{-1}X^Ty = Cy.$$

Thus each $\hat\beta_j$ can be defined linearly in terms of $y_i$, as

$$\hat\beta_j = c_{j0} y_0 + c_{j1} y_1 + c_{j2} y_2 + \cdots,$$

and is therefore a linear estimator. Is there a particular reason we don't consider non-linear estimator, for example, of the form,

$$ \tilde\beta_j = Cy^a = c_{j0} y_0^a + c_{j1} y_1^a + c_{j2} y_2^a + \cdots $$

Improve this question
$\endgroup$
2
  • 4
    $\begingroup$ But we do consider nonlinear estimators, the lasso is a such. $\endgroup$
    – kjetil b halvorsen ♦
    Apr 29, 2017 at 16:20
  • 1
    $\begingroup$ Sorry, my question might not be very clear. I wouldn't count lasso, since (afaik) while it is nonlinear, it is neither unbiased nor lower variance than OLS. Maybe I should rephrase: Are there unbiased, non-linear estimators with lower variance than the OLS estimator? $\endgroup$
    – user126350
    Apr 29, 2017 at 21:51

3 Answers 3

Reset to default
5
$\begingroup$

In mathematics rarely things are developed in the way they're presented in textbooks. That's the real reason. Here's the explanation.

First, someone came up with a problem to fit $$y=X\beta+\varepsilon$$, i.e. find the "best" in some respect set of parameters $\beta$. Whoever did this didn't think that the solution would be a liner combination of $y$'s. He simply thought about what would be the criterion to pick the "best" solution, and came up with minimizing the sum of squared errors $\varepsilon'\varepsilon$. This is a very reasonable criterion for many people. So, he went on and formulated the optimization problem: $$\min_\beta \varepsilon'\varepsilon=\min_\beta(y-X\beta)'(y-X\beta)$$

When the guy solved the problem, he was amazed that the solution turned out to be a linear combination of $y$'s: $$(X'X)^{-1}X'y$$

He wasn't looking for solutions that are BLUE or linear. He was just looking for a solution of least squares problem. Then his friends jumped on to study this solution from different angles and came up with Gauss-Markov theorem, BLUE etc.

After this was all done people today look at all kinds of formulations of "best" solution criteria, they're not simply sums of squared errors anymore. Some people want to also have "small" $\beta$, which leads to all kinds of shrinkage methods that are not BLUE or linear anymore, and so on.

I like your question a lot because it separates out the linear model specification $X\beta$ on independent variables and the fact that the solution is a linear combination of dependent variables $Cy$. In order to come from the latter to the former one needs special kind of goodness-of-fit criteria, such as minimum of sum of squares. Other goodness-of-fit criteria may lead to non-linear (on $y$) solutions.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for this answer! It makes a lot of sense that the solution to choosing the parameters of a linear model will depend on the goodness of fit criteria. On further consideration, I think my real question is then if this problem could have been solved different: Are there unbiased, non-linear estimators with lower variance than the OLS estimator? $\endgroup$
    – user126350
    Apr 29, 2017 at 21:56
  • $\begingroup$ if it's a different question, then post it separately. that way it'll be easier for those who ask the same question in the future. $\endgroup$
    – Aksakal
    Apr 29, 2017 at 22:00
  • $\begingroup$ The OLS solution is simply the best when you have normal errors, among both linear on nonlinear solutions. If you drop the normality then things change, i's not the best anymore, it's only BLUE $\endgroup$
    – Aksakal
    Apr 29, 2017 at 22:04
  • $\begingroup$ I have worked with lowest sum-of-squared relative error, rather than lowest sum-of-squared absolute error. In such cases, "lowest variance" means "lowest relative variance". $\endgroup$
    – James Phillips
    Apr 30, 2017 at 1:42
  • 2
    $\begingroup$ @Aksakal Can you point to a proof that OLS is best "among both linear or nonlinear solutions" with normal errors? This is exactly what I want to know. $\endgroup$
    – user126350
    May 10, 2017 at 16:28
2
$\begingroup$

When the error term is not Gaussian, it will generally be the case that the best estimators (e.g. in terms of MSE) are not linear.

In some cases, all linear estimators may be arbitrarily bad. (It's not always so clear what all the fuss about being BLUE is, when even the best linear estimator may be terrible.)

So for example if the tails of the conditional distribution of the dependent variable are made heavier and heavier, you need give less and less weight to values further away, or the variance of the parameter estimates can be increased beyond any bound.

[Nonlinear estimators include more than just powers, though.]

Improve this answer
$\endgroup$
7
  • $\begingroup$ I'm interested only in the case of Gaussian errors. Certainly, for non-normal errors OLS is not optimal. $\endgroup$
    – user126350
    May 10, 2017 at 16:31
  • $\begingroup$ You make no mention of $\epsilon$ being Gaussian in the question -- indeed, you seem to have just denied the premise of your own question, then; if the error is jointly Gaussian, and the model is linear then ML will be linear; this follows directly from the quadratic form of the exponent in the Gaussian. $\endgroup$
    – Glen_b
    May 10, 2017 at 16:50
  • $\begingroup$ Indeed, I really bungled the original question! Can you elaborate on "the model is linear then ML will be linear; this follows directly from the quadratic form of the exponent in the Gaussian"? What's ML and how does its linearity follow from Gaussian errors? $\endgroup$
    – user126350
    May 10, 2017 at 17:44
  • $\begingroup$ ML=maximum likelihood (sorry). But this is essentially a new question. Consider that in the joint density of errors $\epsilon$, the exponent is $-\frac12\epsilon^\top \Sigma^{-1}\epsilon$ where in ordinary regression $\epsilon=(y-X\beta)$, and $\Sigma=\sigma^2I$. Then $-2\log L_\beta = k+c_\sigma(y-X\beta)^\top(y-X\beta)$, and (handwaving detail in relation to $\sigma^2$) the argmax of $L$ is the minimizer of $(y-X\beta)^\top(y-X\beta) = y^\top y -2\beta^\top X^\top y + \beta^\top X^\top X\beta$. If we take the derivative with respect to $\beta$, setting=0 and solving for $\hat\beta$ ...ctd $\endgroup$
    – Glen_b
    May 10, 2017 at 22:44
  • $\begingroup$ ctd ... we get the linear estimator $\hat{\beta}=(X^\top X)^{-1}X^\top y$. Similar results apply for other choices of $\Sigma$ as long as $\epsilon$ is linear in $\beta$ and $y$. e.g. you could generalize to some unspecified $\epsilon=B y - A\beta$ for some conformal matrices $B$ and $A$ (presumably some function of a set of predictors). When you expand $-2\log L$ the term $\epsilon^\top \Sigma^{-1}\epsilon$ has a term in $\beta^\top My$ (or equivalently in $y^\top M^\top \beta$) & a term like $\beta^\top Q_\Sigma \beta$ and so we end up with $\hat\beta=Q_\Sigma^{-1}My$ $\endgroup$
    – Glen_b
    May 10, 2017 at 22:49
0
$\begingroup$

The question is how you find $\beta$? How do you actually derive OLS? You differentiate according to a certain parameter, and you get a set of equations, which you then solve. If $\beta$ is not linear in the model equation, the solution will not be a linear combination of $Y$. (It will probably not be a linear combination of a transformation of $Y$ either - which is what your $\widetilde{\beta}$ is) It might not have a close form, and can only be approximated numerically.

Next - how do you prove your estimator is the best (i.e. BLUE)? Linearity of Expectations, and linear operations on Variance - are usually needed to say anything about an estimator. You will usually need to use Linearity of Expectations to show that the estimator is unbiased, and you would need to use linear operations on Variance to show that it is minimal. This is what Gauss-Markov theorem does. But if your model is not linear in parameters, the estimator won't be linear in $Y$, so most likely you won't be able to use these.

So, it's not that you don't consider non-linear models - it's just that you need a way to a. find the estimator, b. show it's good.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.