4
$\begingroup$

I looked at robust regression for the first time today and I am a bit confused, comparing it to something like ordinary least squares and I am not sure if I am on the right track.

I read a few articles and they say that with robust regression you don't need to worry too much about outliers and heteroscedasticy and the normality restrictions on the residuals is not as important as with OLS. When I do robust regression in R however, I also don't get any significant indicators regarding coefficients (I am using the rlm() function). So when robust regression like M or MM-estimation is applied in a regression model, is the significance of coefficients still important or is the idea of robust regression just to fit the best possible plane through the data points and find the coefficients?

$\endgroup$
  • 3
    $\begingroup$ use the robustbase package instead. Its more more recent and contains the information (t statistics) you seek. $\endgroup$ – user603 Apr 23 '14 at 19:34
3
$\begingroup$

To be somewhat nitpicky, I would not quite say that outliers, heteroscedasticity, and non-normality don't matter with robust regression methods. Rather, I would say that robust methods are less likely to be impaired or harmed by those conditions. However, they could still have a negative effect.

The issue of whether the significance of the coefficients or the accuracy of their estimation is what's important is really unrelated to robust regression. Which of those is more important to you depends on the questions you are trying to answer, not what tools you use to try to answer them. Instead, consider a case where you want to test the hypothesis that a given variable is unrelated to the response variable. You wouldn't want the answer you get to that question (either yes or no) to be driven by an outlier. So you would use robust methods to help ensure that your answer is representative of the bulk of your data. Likewise, consider a case where you want to know the slope of the relationship between a predictor variable and the response variable as accurately as possible. You wouldn't want the estimated slope value that you get to have been driven by an outlier. So you would use robust regression to protect against that possibility. In short, robust methods diminish the extent to which your results might be influenced by violations of the classical statistical assumptions.

I recognize your frustration that you did not get any significant results when you used these methods. There are a couple of possibilities here. It may be that what appeared to be the case prior to using robust regression (perhaps the results from a prior OLS regression analysis) were driven by violations of the OLS assumptions and the null hypothesis is actually true. The other possibility is that, when OLS assumptions do hold, standard methods will have more power than robust methods.

$\endgroup$
2
$\begingroup$

With OLS regression, when all the assumptions hold, then the t-statistics will follow a t distribution (when the null is true) and the p-value is straight forward to compute.

With the robust regressions it is no longer straight forward and we don't really know what the distribution of the "t" statistic is, so it is better to not provide a p-value then to provide one that is most likely wrong (but may be trusted if seen). You can still use the rule of thumb that "t" statistics near 0 are probably not significant and ones far from 0 are probably significant, but we don't really know where the divide is between "close" and "far". One option for producing p-values is to bootstrap the process, see the boot package for tools that will help with this.

$\endgroup$
  • 1
    $\begingroup$ you don t want to use a generic bootstrap package to bootstrap robust estimates. There are enormously more efficient ways to get bootstrapped CI's (computer wize) for robust estimated. Have a look at the frb package and this joss paper $\endgroup$ – user603 Apr 24 '14 at 6:55
  • 1
    $\begingroup$ @user603 In many cases using a generic package is not only sufficient but the better choice. It's not worth it to spend some hours of a scientist just to shorten a computer calculation by a few minutes. Only when you start doing very similar things a lot of time it is worth to look at efficiency. $\endgroup$ – Erik Apr 24 '14 at 7:02
  • $\begingroup$ @Erik: you might want to read the JOSS paper before commenting. We're talking of 2 orders of magnitude saving in computing times with no compromise on the validity of the bootstraped results. $\endgroup$ – user603 Apr 24 '14 at 9:15
  • 1
    $\begingroup$ @user603 A saving in two orders of magnitude only matters if the computing time is significant to start with. If I have to do it once for a paper I don't care if the calculation takes 5 minutes or 5 seconds. My time is much more valuable than that of the computer. $\endgroup$ – Erik Apr 24 '14 at 9:26
  • 1
    $\begingroup$ @user603 What do I care about exponential time complexity in the number of samples when I never have more than say 10? Listen, there are times and places where execution speed matters and I once wrote myself a fast C function to bootstrap AUCs of classifiers since I had to do it for a lot of them. But those are special cases. I don't think using special tools for everything is good general advice. And I think we should stop this conversation now, since it does not really contribute to the answer anymore. $\endgroup$ – Erik Apr 24 '14 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.