$L_2$ minimisation has a very nice geometric explanation as projection onto a subspace of the appropriate size. Is there a similar "real explanation" for any of the approaches to robust regression?

Edit: For example Huber's M estimates of regression and Yohai's MM estimator of regression.

  • $\begingroup$ I think that you ought to define and even outline, the robust regression. For there exist not one type of "robust" regression. Reader would prefer your description/synopsis to reading an external document. $\endgroup$ – ttnphns Mar 21 '15 at 6:14
  • $\begingroup$ @ttnphns How about this: is there a geometric description of any of the robust regression models? $\endgroup$ – isomorphismes Mar 21 '15 at 17:54
  • $\begingroup$ @isomorphismes the problem is that "robust" is a generic term. But based on the documentation for rlm, it seems like you're asking about regression with robust standard errors like the Huber-White estimator. If you can clarify, this will be a good question and I'll be interested in the answer. $\endgroup$ – shadowtalker Mar 21 '15 at 18:37
  • 1
    $\begingroup$ @user603 I must have misread the docs then, my mistake. I skimmed them and saw "white estimator" $\endgroup$ – shadowtalker Mar 22 '15 at 13:36
  • 1
    $\begingroup$ @ssdecontrol Feel free to write about the geometry of whitened errors too. I've tried to broaden the question to allow as many interesting answers as possible. $\endgroup$ – isomorphismes Mar 22 '15 at 15:03

Yes! Robust regression has a clear geometric interpretation.

One can think about the geometry of an estimator by looking at the group of equivariance to which it belongs. Quick example;

Example scale estimator $S(x)$ (the usual variance, $\sigma^2(x)$, and the median absolute deviation, $\mbox{mad}(x)$, are two cart bearing members of this group) are equivariant to multiplications of the data by a constant:

$$S(\alpha x)=|\alpha|S(x),\quad\alpha\in\mathbb{R}$$

In other words, the group of equivariance defines the transformations of the data which, in some sense, you don't need to care about when using the estimator because when such a transformation is applied to the data, the estimator changes with the data 'in the natural way'.

These groups of equivariance also have bearing on important properties of the estimators such as consistency.

Likewise, a regression estimator $T(\pmb x,y)\in\mathbb{R}^p$ is characterized by at least two group of equivariance:

  • $T(\pmb x,y)$ is regression equivariant: $$T(\pmb x,y+\pmb\beta'x)=T(\pmb x,y)+\pmb\beta,\quad\pmb\beta\in\mathbb{R}^p$$
  • $T(\pmb x,y)$ is affine equivariant: $$T(\pmb x\pmb A,y)=\pmb A^{-1}T(\pmb x,y)$$ for any non singular matrix $\pmb A\in\mathbb{R}^{p\times p}$. This implies that $T(\pmb x,y)$ is residual admissible: the regression estimates only depend on the data through the vector of residuals.

The regression estimator estimated by rlm, like the usual OLS estimators, all satisfy affine and regression equivariance.

Note that there exist some robust regression estimators that belong to group of equivariance to which OLS doesn't belong (e.g. that have in this sense a stronger geometry than OLS). Think of invariance to monotone transformations that holds for quantile-based estimators such as the $\mbox{mad}(x)$ (and in the case of monotone transformation of the responses for quantile regression) but not for the variance (or the usual OLS estimators).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.