29
$\begingroup$

In linear regression analysis, we analyze outliers, investigate multicollinearity, test heteroscedasticty.

The question is: Is there any order to apply these? I mean, do we have to analyze outliers very firstly, and then examine multicollinearity? Or reverse?

Is there any rule of thumb about this?

$\endgroup$
3
  • 3
    $\begingroup$ Some very rough rules of thumb: you should investigate collinearity before you do any fitting. If you find it is present, you should either (a) use a method that handles collinearity, (b) remove collinear features, or (c) transform your features (eg using PCA). Once you have fitted a model, you can look for heteroscedasticity in the residuals. In general, if you are making a predictive model you should not remove outliers. Instead, use a method which is robust to the presence of outliers. $\endgroup$ Jul 19, 2012 at 7:32
  • 1
    $\begingroup$ How does one best investigate collinearity? Looking at the off-diagonal elements of the predictors' correlation matrix? $\endgroup$
    – miura
    Jul 19, 2012 at 8:10
  • 2
    $\begingroup$ The best way to investigate collinearity is condition indices and proportion of variance explained by them. High correlation is neither a necessary nor a sufficient condition for collinearity. $\endgroup$
    – Peter Flom
    Jul 25, 2012 at 17:21

2 Answers 2

34
$\begingroup$

The process is iterative, but there is a natural order:

  1. You have to worry first about conditions that cause outright numerical errors. Multicollinearity is one of those, because it can produce unstable systems of equations potentially resulting in outright incorrect answers (to 16 decimal places...) Any problem here usually means you cannot proceed until it is fixed. Multicollinearity is usually diagnosed using Variance Inflation Factors and similar examination of the "hat matrix." Additional checks at this stage can include assessing the influence of any missing values in the dataset and verifying the identifiability of important parameters. (Missing combinations of discrete independent variables can sometimes cause trouble here.)

  2. Next you need to be concerned whether the output reflects most of the data or is sensitive to a small subset. In the latter case, everything else you subsequently do may be misleading, so it is to be avoided. Procedures include examination of outliers and of leverage. (A high-leverage datum might not be an outlier but even so it may unduly influence all the results.) If a robust alternative to the regression procedure exists, this is a good time to apply it: check that it is producing similar results and use it to detect outlying values.

  3. Finally, having achieved a situation that is numerically stable (so you can trust the computations) and which reflects the full dataset, you turn to an examination of the statistical assumptions needed for correct interpretation of the output. Primarily these concerns focus--in rough order of importance--on distributions of the residuals (including heteroscedasticity, but also extending to symmetry, distributional shape, possible correlation with predicted values or other variables, and autocorrelation), goodness of fit (including the possible need for interaction terms), whether to re-express the dependent variable, and whether to re-express the independent variables.

At any stage, if something needs to be corrected then it's wise to return to the beginning. Repeat as many times as necessary.

$\endgroup$
11
  • 2
    $\begingroup$ I actually prefer to use condition indices rather than VIFs. I did my dissertation on these, a while back. $\endgroup$
    – Peter Flom
    Jul 25, 2012 at 17:23
  • 1
    $\begingroup$ @Peter Good point. I prefer condition indices, too, but it seems to me that VIFs are very popular now. $\endgroup$
    – whuber
    Jul 25, 2012 at 18:17
  • $\begingroup$ whuber, I followed here from your comment earlier today. I once consulted with a statistician during my postdoc about some concerns regarding multicollinearity. He professed a view that, depending on the nature of the IVs in a regression, collinearity could be considered structurally part of the phenomena being modeled. I am probably mangling his precise language, and I would have to dig to even find his name again, but do you know any texts that would motivate a nuanced reasoning about multicollinearity along these lines? Just an off-chance ask. :) $\endgroup$
    – Alexis
    Mar 11, 2018 at 0:31
  • 1
    $\begingroup$ @whuber perhaps there are various definitions floating around; when I google'd "condition indices", I was lead to this NIST page itl.nist.gov/div898/software/dataplot/refman2/auxillar/… which defines them as the squared singular values of the normalized design matrix, which is exactly exactly equivalent to a PCA varcomp ;). I agree that the concept of a condition number is distinct from these condition indices so-defined. $\endgroup$ Jan 16, 2023 at 19:43
  • 1
    $\begingroup$ @John I suspect Peter F intended to refer to condition numbers. Those are more directly relevant to analysis of multicollinearity. That was how I understood his term when I responded ten years ago. $\endgroup$
    – whuber
    Jan 16, 2023 at 19:48
4
$\begingroup$

I think it depends on the situation. If you don't expect any particular problems you can probably check these in any order. If you expect outliers and might have a reason to remove them after detecting them then check for outliers first. The other issues with the model could change after observations are removed. After that the order between multicollinaerity and heteroscedasticity doesn't matter. I agree with Chris that outliers should not be removed arbitrarily. You need to have a reason to think the observations are wrong.

Of course if you observe multicollinearity or heteroscedasticity you may need to change your approach. The multicollinearity problem is observed in the covariance matrix but there are specific diagnostic tests for detecting multicollinearity and other problems like leverage points look at the Regression Diagnostics book by Belsley, Kuh and Welsch or one of Dennis Cook's regression books.

$\endgroup$
1
  • 9
    $\begingroup$ Michael, In the future, can you use the formatting options? (the correct key to insert links is ctrl-l, not ctrl-c). $\endgroup$
    – user603
    Jul 19, 2012 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.