4
$\begingroup$

Is there a general rule of thumb about when robust regression or quantile regression is preferred in the presence of outliers?

For example, I have a dataset where the DV exhibits extreme positive skewness. However, the large cases are actually some of the most interesting observations. When I run OLS, I find a positive relation between the DV and the IV of interest. When I estimate quantile regressions, I find that the positive relation between the DV and IV of interest is strongest in the 85th, 90th, and 95th percentiles (which is where one might expect). It is insignificant and sometimes negative for the rest of the percentiles. However, when I run rreg in Stata, it basically gives no weight to the large positive outliers, leading to no relation between the DV and the IV of interest. Which approach (OLS, Quantile, rreg) should be reported? Which is most appropriate?

Improve this question
$\endgroup$
8
  • $\begingroup$ This can't be answered for your data because (1) your verbal description isn't enough to convey what they look like (2) we can't know your scientific context implying what kind of model matches your goals. It can't be answered generally: any survey of robust regression shows numerous competing methods; people can't agree that one is best and there are many good reasons why that is inevitable. It could be that quite another answer makes as much or more sense, namely use a transformation or non-identity link function rather than try to squeeze your data into hyperplane + long-tailed errors. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 9:59
  • 1
    $\begingroup$ Note that rreg in Stata should be expected to mean precisely nothing to non-Stata users. It's an implementation of the method of Li, G. 1985. Robust regression. In Exploring Data Tables, Trends, and Shapes, ed. D. C. Hoaglin, F. Mosteller, and J. W. Tukey, 281-340. New York: Wiley. I'd be surprised if anyone regards that as the method of choice in 2013. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 10:04
  • 1
    $\begingroup$ I note that further that whether robust and quantile regression are alternatives appears moot. I guess many people would want a clearer idea of your precise goals before recommending one or the other. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 10:07
  • $\begingroup$ DV is bounded between 0 and 1, with a lot of cases close to zero, approx 30% of the cases > 0.001, and approximately 1% of the cases > 0.01. So extreme positive skew. My precise goals are to evaluate the relation between DV and IV of interest. However, OLS seems inappropriate given skewness. Two methods of dealing with skewness - Quantile and Robust Reg, give different results. One shows pos & sig relation between IV and DV in right tail (which economics would suggest), robust reg discards DVs in right tail, leading to insigifnicant relation. Which should I choose, given different results? $\endgroup$
    – KSL
    Nov 28, 2013 at 14:51
  • 1
    $\begingroup$ From the first sentence alone I would not expect any method that fits a line (plane, hyperplane) to work well with your data: differences between robust and quantile are a separate issue. None of the methods so far discussed pays any attention to boundedness of the response. If you don't have exact zeros you might, just possibly, benefit from transformation; otherwise some generalised linear model might help. You are seeking oracular judgements on what is right or what is better that can't be given responsibly when you are only gradually revealing vital characteristics of your data. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 14:57

1 Answer 1

Reset to default
4
$\begingroup$

Is there a general rule of thumb about when robust regression or quantile regression is preferred in the presence of outliers?

Yes. So long as we're comparing regression equivariant approaches, it is clearly possible to rank the various robust estimates of regression in terms of their capacity to find outliers.

The algorithm behind rreg is described here:

rreg first performs an initial screening based on Cook’s distance $>1$ to eliminate gross outliers before calculating starting values and then performs Huber iterations followed by biweight iterations, as suggested by Li

The Li estimate of regression is in a sense similar to an S-estimator but with a single starting point. This estimator is not used a lot and has not been studied much. I would advise you to use the FastS algorithm of Saliban-Barrera&Yohai, about which much more is known.

For more background on why the S-estimator, a robust estimator with re-descending $\rho$ function, is more reliable than quantile regression check this answer. The S-estimates of regression are implemented in Stata, check the Verardi and Croux (2008) stata package and companion paper.

For the second part of your question: the breakdown point of quantile regression is proportional to the quantile you estimate with it. So the $\tau=0.9$ quantile of the quantile regression is much less able to withstand outliers than the $\tau=0.5$ quantile (and is generally not considered robust).

By the way, the fact that an observation is flagged as an outlier does not imply anything about the quality, validity or reliability of the corresponding measurement. It simply means that the flagged observation is inconsistent with the multivariate pattern fitting the bulk of the data. Indeed, in many fields (micro-array analysis, fraud identification) revealing such data points is often the primary objective of the study.

[1]Verardi, Croux (2008). Robust regression in Stata. The Stata Journal 9(3): 439-453.
[2]Salibian-Barrera M., Yohai, V.J. (2006). A Fast Algorithm for S-Regression Estimates. Journal of Computational and Graphical Statistics, Vol. 15, 414--427.

Improve this answer
$\endgroup$
16
  • 2
    $\begingroup$ stata.com/manuals13/rrreg.pdf gives a much better description of rreg in Stata. The account at the UCLA website omits many important details, even for a verbal sketch. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 10:29
  • 2
    $\begingroup$ if you want to estimate a given conditional quantile, then, yes, the quantile regression will do that for you. But this is neither an outlier detection tool nor a robust fitting procedure (I think the tags you placed on your question are misleading). $\endgroup$
    – user603
    Nov 28, 2013 at 14:45
  • 2
    $\begingroup$ NB: OLS is an estimation procedure, not a model. Please don't conflate the two. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 15:00
  • 2
    $\begingroup$ How can quantile regression far into the tails be robust? It is designed to return whatever quantile you ask for. That is why I emphasised earlier that robust and quantile regression are not to be seen as alternatives. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 18:42
  • 2
    $\begingroup$ Again, sorry, but just having some experience in data analysis doesn't impart to me the ability to act as your oracle. What you describe could be what you summarize it as, or it could be a side-effect of fitting on the wrong scale. If your data are piled up near zero on your response scale, then necessarily it is hard to distinguish the lower quantiles from close parallels to the x axis. $\endgroup$
    – Nick Cox
    Nov 28, 2013 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.