3
$\begingroup$

From my question here, it is evident that estimation approaches to linear regression other than ordinary least squares can result in the predictions and residuals lacking orthogonality, despite the model being linear.

What, if any, approaches to estimating the $\beta$ of $y=X\beta+\epsilon$ are there that are not equivalent to ordinary least squares (yield a different answer than $\hat\beta=(X^TX)^{-1}X^Ty$) yet still yield this orthogonality?

Let’s rule out $\hat\beta=\vec 0$. If it happens that $X\hat\beta=X\hat\beta_{ols}$, so be it, but that is not a requirement.

Improve this question
$\endgroup$
5
  • $\begingroup$ I think OLS is the unique estimator which minimizes any weighted sum of squares which has this property. (due to uniqueness of the orthogonal projection operator). $\endgroup$
    – John Madden
    Commented Oct 3, 2022 at 23:22
  • $\begingroup$ You can always adjust the predictions of any other estimator by adding a constant. In most cases the constant can be chosen to make the orthogonality condition hold. The size of this constant is $O(n^{-1}).$ $\endgroup$
    – whuber ♦
    Commented Oct 3, 2022 at 23:33
  • 1
    $\begingroup$ @whuber That’s interesting, but it seems to contradict what Ben posted. Perhaps you could expand on that in an answer, please. $\endgroup$
    – Dave
    Commented Oct 4, 2022 at 0:21
  • $\begingroup$ It's not a contradiction, because the prediction does not comprise the entire column space of the design matrix. $\endgroup$
    – whuber ♦
    Commented Oct 4, 2022 at 13:38
  • $\begingroup$ Could you clarify what you mean by "lacking orthogonality"? After all, you could (artificially and completely arbitrarily) construct a different design matrix $Z$, use it to compute another estimator $\tilde y$ using OLS, and $\tilde y$ would satisfy the equations on your linked question. This $\tilde y$ would usually not be the least squares estimator associated with $X,$ of course! $\endgroup$
    – whuber ♦
    Commented Oct 4, 2022 at 22:28

1 Answer 1

Reset to default
1
$\begingroup$

For the benefit of completeness, I'll note that you have ruled out the case where $\hat{\boldsymbol{\beta}}=\mathbf{0}$, which gives the zero vector for the predicted response. The zero vector is (trivially) orthogonal to any other vector, so this estimator also gives a predicted response vector that is (trivially) orthogonal to the residual vector.

Setting aside this special case, or mixtures with this case, the hat matrix from OLS is the unique projection matrix that gives an orthogonal projection onto the column space of the design matrix. This projection uniquely defines OLS estimation (equivalently MLE under the Gaussian linear model) and so there are no other estimation methods that use this orthogonal projection that are not equivalent to OLS. This means that the only way you can get a guarantee of (non-trivial) orthogonality is to use OLS estimation.

Improve this answer
$\endgroup$
12
  • $\begingroup$ I’d like to think this is true, but what about $\hat\beta=\vec 0?$ Is that just the “it’s orthogonal to everything” zero vector solution? Then what about the comment by @whuber related to the $O(n^{-1})$ constant? $\endgroup$
    – Dave
    Commented Oct 4, 2022 at 0:21
  • $\begingroup$ @Dave: The estimator you give is ruled out in the text of the question. Re the comment by whuber, I take the requirement to "guarantee" orthogonality to mean that orthogonality must occur under every possible dataset. $\endgroup$
    – Ben
    Commented Oct 4, 2022 at 1:21
  • 1
    $\begingroup$ @Dave: While it is outside the scope of the question, if you use the estimator $\hat{\beta}=0$ then the predicted response vector is also the zero vector, which is trivially orthogonal to any other vector (including the residual vector). Re proof of the remaining case, it can be constructed by noting that a guarantee of orthogonality requires the use of an orthogonal projection onto the column-space of the design matrix (see e.g., this related question) and this projection is uniquely defined. $\endgroup$
    – Ben
    Commented Oct 4, 2022 at 2:07
  • 1
    $\begingroup$ @Dave: I mean that you could create an estimator that is the OLS estimator with some arbitrary probability $\phi(\mathbf{x},\mathbf{y})$ and the zero vector with probability $1-\phi(\mathbf{x},\mathbf{y})$ and that would still guarantee orthogonality (again, relying on trivial orthogonality in the second case). $\endgroup$
    – Ben
    Commented Oct 4, 2022 at 8:20
  • 1
    $\begingroup$ @Ben Interesting, I hadn’t thought of it like that. $\endgroup$
    – Dave
    Commented Dec 5, 2022 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.