Robust regression: setting the limit between errors and influential observations

Question

This thread discusses an issue found in robust regression estimation: when the model fit is particularly low, although significant, the Huber estimator with a standard $k$ value of, say, 2.5, downweights most of the observations. The result is very low coefficients and, loosely speaking, under-estimation of the predicted values.

When the errors dispersion is relatively compact and the model fit low, like in this example, which robust estimator is more efficient? Does it make sense to set Huber $k$ to some arbitrary errors quantile?

These are the diagnostics from the ordinary linear regression (lm). Note how Cook's distance is not flagging any particular influential point:

Answer 1

Your QQ plot bears a strong resemblance to that of a t distribution with 2 degrees of freedom (plot based on 40000 observations):

A t(3) and a Cauchy with sample sizes of 40,000 don't look as much like your QQ plot as the t(2), but note I'm not saying this is evidence that your error distribution is a t(2) or is even well-approximated by one. The point is that your distribution is very fat-tailed indeed.

With the t family, the optimal estimator of location is redescending, meaning, heuristically, that the weight applied to extreme observations goes to zero faster than the observation value goes to infinity. This means that extreme values receive less weight than with the Huber estimator, for which absolute values greater than the parameter $k$ have weights that, in effect, go to zero as fast as the observation value goes to infinity. In the Wikipedia page for "Robust statistics", a little over halfway down, is a graphical comparison of the influence of values on the Huber (called "Winsorizing at 1.5") and Tukey biweight (a redescending estimator.) A little farther down is a plot of the influence function of the optimal estimator for various t distributions.

Consequently, with this data, I'd not use the Huber $\psi$-function at all, instead setting psi=biweight in your call to rlm, and accepting the fact that a lot of your observations will have little or no influence on the final estimates. Note, as @Macro observed in the thread to the other question, that this isn't a question of identifying and downweighting outliers, this is actually trying to get somewhat close to optimal estimates given your errors really do come from a very fat-tailed distribution - no outliers required or assumed.

Answer 2

This looks like a case where a linear fit might not be a good idea and there are certainly outliers and leverage points. In your first plot it looks like your estimator is tring to fit just 3 or 4 point very closely and is essentially ignoring several outliers. But I don't think it is fitting three points and ignoring the rest. It appears to me that it is trying to go through te center of that oval shape mass of points and coincidentally by doing that it practically runs through three or four isolated points. The Q-Q plot shows heavytailed behavior because of the large residuals both positive and negative due to the outliers. That should not be a surprise. What I find interesting is that the robust regression gives a negative slope to the line. it would be helpful to see the scatter plot of X vs Y so we could imagin what the least squares estimate would be. If there are leverage points for extreme values of x the slope could even change sign. Maybe any linear model would not help this. It could be that X and Y are practically uncorrelated.