In linear regression analysis, we analyze outliers, investigate multicollinearity, test heteroscedasticty.

The question is: Is there any order to apply these? I mean, do we have to analyze outliers very firstly, and then examine multicollinearity? Or reverse?

Is there any rule of thumb about this?

  • 2
    $\begingroup$ Some very rough rules of thumb: you should investigate collinearity before you do any fitting. If you find it is present, you should either (a) use a method that handles collinearity, (b) remove collinear features, or (c) transform your features (eg using PCA). Once you have fitted a model, you can look for heteroscedasticity in the residuals. In general, if you are making a predictive model you should not remove outliers. Instead, use a method which is robust to the presence of outliers. $\endgroup$ – Chris Taylor Jul 19 '12 at 7:32
  • 1
    $\begingroup$ How does one best investigate collinearity? Looking at the off-diagonal elements of the predictors' correlation matrix? $\endgroup$ – miura Jul 19 '12 at 8:10
  • 1
    $\begingroup$ The best way to investigate collinearity is condition indices and proportion of variance explained by them. High correlation is neither a necessary nor a sufficient condition for collinearity. $\endgroup$ – Peter Flom Jul 25 '12 at 17:21

The process is iterative, but there is a natural order:

  1. You have to worry first about conditions that cause outright numerical errors. Multicollinearity is one of those, because it can produce unstable systems of equations potentially resulting in outright incorrect answers (to 16 decimal places...) Any problem here usually means you cannot proceed until it is fixed. Multicollinearity is usually diagnosed using Variance Inflation Factors and similar examination of the "hat matrix." Additional checks at this stage can include assessing the influence of any missing values in the dataset and verifying the identifiability of important parameters. (Missing combinations of discrete independent variables can sometimes cause trouble here.)

  2. Next you need to be concerned whether the output reflects most of the data or is sensitive to a small subset. In the latter case, everything else you subsequently do may be misleading, so it is to be avoided. Procedures include examination of outliers and of leverage. (A high-leverage datum might not be an outlier but even so it may unduly influence all the results.) If a robust alternative to the regression procedure exists, this is a good time to apply it: check that it is producing similar results and use it to detect outlying values.

  3. Finally, having achieved a situation that is numerically stable (so you can trust the computations) and which reflects the full dataset, you turn to an examination of the statistical assumptions needed for correct interpretation of the output. Primarily these concerns focus--in rough order of importance--on distributions of the residuals (including heteroscedasticity, but also extending to symmetry, distributional shape, possible correlation with predicted values or other variables, and autocorrelation), goodness of fit (including the possible need for interaction terms), whether to re-express the dependent variable, and whether to re-express the independent variables.

At any stage, if something needs to be corrected then it's wise to return to the beginning. Repeat as many times as necessary.

  • 2
    $\begingroup$ I actually prefer to use condition indices rather than VIFs. I did my dissertation on these, a while back. $\endgroup$ – Peter Flom Jul 25 '12 at 17:23
  • 1
    $\begingroup$ @Peter Good point. I prefer condition indices, too, but it seems to me that VIFs are very popular now. $\endgroup$ – whuber Jul 25 '12 at 18:17
  • $\begingroup$ whuber, I followed here from your comment earlier today. I once consulted with a statistician during my postdoc about some concerns regarding multicollinearity. He professed a view that, depending on the nature of the IVs in a regression, collinearity could be considered structurally part of the phenomena being modeled. I am probably mangling his precise language, and I would have to dig to even find his name again, but do you know any texts that would motivate a nuanced reasoning about multicollinearity along these lines? Just an off-chance ask. :) $\endgroup$ – Alexis Mar 11 '18 at 0:31
  • $\begingroup$ @Alexis It sounds like that statistician has a nuanced and sophisticated concept of multicollinearity. I am unable to think of any textbook that clearly expresses it. $\endgroup$ – whuber Mar 11 '18 at 14:25
  • $\begingroup$ I will just have to track him down and ask him about it. :) $\endgroup$ – Alexis Mar 11 '18 at 15:56

I think it depends on the situation. If you don't expect any particular problems you can probably check these in any order. If you expect outliers and might have a reason to remove them after detecting them then check for outliers first. The other issues with the model could change after observations are removed. After that the order between multicollinaerity and heteroscedasticity doesn't matter. I agree with Chris that outliers should not be removed arbitrarily. You need to have a reason to think the observations are wrong.

Of course if you observe multicollinearity or heteroscedasticity you may need to change your approach. The multicollinearity problem is observed in the covariance matrix but there are specific diagnostic tests for detecting multicollinearity and other problems like leverage points look at the Regression Diagnostics book by Belsley, Kuh and Welsch or one of Dennis Cook's regression books.

  • 9
    $\begingroup$ Michael, In the future, can you use the formatting options? (the correct key to insert links is ctrl-l, not ctrl-c). $\endgroup$ – user603 Jul 19 '12 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.