3
$\begingroup$

This thread discusses an issue found in robust regression estimation: when the model fit is particularly low, although significant, the Huber estimator with a standard $k$ value of, say, 2.5, downweights most of the observations. The result is very low coefficients and, loosely speaking, under-estimation of the predicted values.

When the errors dispersion is relatively compact and the model fit low, like in this example, which robust estimator is more efficient? Does it make sense to set Huber $k$ to some arbitrary errors quantile?

These are the diagnostics from the ordinary linear regression (lm). Note how Cook's distance is not flagging any particular influential point:

Robust regression diagnostics

|cite|improve this question
$\endgroup$
  • $\begingroup$ What is your response variable? How is it measured? $\endgroup$ – whuber♦ Jun 1 '12 at 16:06
  • $\begingroup$ @whuber The model is trying to predict seismic movements every 10 seconds, based on a number of real-time predictors. $\endgroup$ – Robert Kubrick Jun 1 '12 at 16:14
  • $\begingroup$ Yes, but how exactly is a "seismic movement" measured and represented? On the face of it, such a movement would (in its simplest form) be a 3D vector. You have also mentioned an interesting and important factor: your data must be time series. That suggests there may be strong autocorrelations among the predictors and predictand, which is something you can (and need to) exploit. $\endgroup$ – whuber♦ Jun 1 '12 at 17:39
  • $\begingroup$ @whuber About the time series, I've tried to use lagged versions of the predictors (around 0.10 correlation for some) but they don't give me a significant prediction improvement. The seismic measure is represented by a linear scale calculated aggregating a number of factors. But in general my question is about robust regression and the Huber estimator when facing a low fit model. $\endgroup$ – Robert Kubrick Jun 1 '12 at 17:51
3
$\begingroup$

Your QQ plot bears a strong resemblance to that of a t distribution with 2 degrees of freedom (plot based on 40000 observations):

enter image description here

A t(3) and a Cauchy with sample sizes of 40,000 don't look as much like your QQ plot as the t(2), but note I'm not saying this is evidence that your error distribution is a t(2) or is even well-approximated by one. The point is that your distribution is very fat-tailed indeed.

With the t family, the optimal estimator of location is redescending, meaning, heuristically, that the weight applied to extreme observations goes to zero faster than the observation value goes to infinity. This means that extreme values receive less weight than with the Huber estimator, for which absolute values greater than the parameter $k$ have weights that, in effect, go to zero as fast as the observation value goes to infinity. In the Wikipedia page for "Robust statistics", a little over halfway down, is a graphical comparison of the influence of values on the Huber (called "Winsorizing at 1.5") and Tukey biweight (a redescending estimator.) A little farther down is a plot of the influence function of the optimal estimator for various t distributions.

Consequently, with this data, I'd not use the Huber $\psi$-function at all, instead setting psi=biweight in your call to rlm, and accepting the fact that a lot of your observations will have little or no influence on the final estimates. Note, as @Macro observed in the thread to the other question, that this isn't a question of identifying and downweighting outliers, this is actually trying to get somewhat close to optimal estimates given your errors really do come from a very fat-tailed distribution - no outliers required or assumed.

|cite|improve this answer
$\endgroup$
1
$\begingroup$

This looks like a case where a linear fit might not be a good idea and there are certainly outliers and leverage points. In your first plot it looks like your estimator is tring to fit just 3 or 4 point very closely and is essentially ignoring several outliers. But I don't think it is fitting three points and ignoring the rest. It appears to me that it is trying to go through te center of that oval shape mass of points and coincidentally by doing that it practically runs through three or four isolated points. The Q-Q plot shows heavytailed behavior because of the large residuals both positive and negative due to the outliers. That should not be a surprise. What I find interesting is that the robust regression gives a negative slope to the line. it would be helpful to see the scatter plot of X vs Y so we could imagin what the least squares estimate would be. If there are leverage points for extreme values of x the slope could even change sign. Maybe any linear model would not help this. It could be that X and Y are practically uncorrelated.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Michael, the plots are for the non-robust linear regression lm() to show the residuals without downweighting. $\endgroup$ – Robert Kubrick Jun 1 '12 at 17:16
  • $\begingroup$ Sorry I read too quickly. I am a little surprised that the red line represents the least squares fit. I would have thought that the outlier with the huge residual of 50 at a fitted value of ten would have strong enough leverage to pull the curve to a positive slope without much change on the residuals in the central mass. But maybe it is just that my mind wants to view this as a scatter plot where the slope of the line would represent the slope parameter estimate for the model. But it is not. $\endgroup$ – Michael Chernick Jun 1 '12 at 17:36
  • $\begingroup$ The abscissa is actually the fitted Ys and the residuals don't tell me what the Xs are. The slope for the X covariate is not depicted there. $\endgroup$ – Michael Chernick Jun 1 '12 at 17:37
  • $\begingroup$ @RobertKubrick Is the lower plot with the V shaped fitted Ys the result for the robust regrssion? $\endgroup$ – Michael Chernick Jun 1 '12 at 17:39
  • $\begingroup$ Nope, all the diag plots are from lm(). $\endgroup$ – Robert Kubrick Jun 1 '12 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.