2
$\begingroup$

my supervisor has given me a response variable that is a count variable, along with 15 explanatory variables. Since the response is a count, I was thinking of doing either Poisson or Negative Binomial regression on the response. However, after computing correlation table and the VIFs, I realized that some of the explanatory variables are highly correlated. I asked my supervisor if I can get rid of some of the explanatory variables in order to avoid the multicollinearity problem, however he said no. How can I safely impose GLM to a data that has high collinearity (some of the correlation coefficients are over .90)?

$\endgroup$
  • $\begingroup$ The problem with high correlation among regressors is... $\endgroup$ – tchakravarty Dec 29 '13 at 19:39
  • $\begingroup$ ....multicolinearity? $\endgroup$ – Jin-Dominique Dec 29 '13 at 19:48
  • 1
    $\begingroup$ Unless you are working with strictly orthogonal regressors, they are always going to be dependent. What then is multicollinearity, and why would you think that the estimate of the variance that you get is not the correct one given your data generating process? You might be confusing numerical analysis with statistical analysis. $\endgroup$ – tchakravarty Dec 29 '13 at 19:51
  • 1
    $\begingroup$ It doesn't make anything large. The estimated variance of the estimator is large. You should not choose a different model because of your estimator's properties. Get a different estimator if you so wish but keep the tradeoffs among estimators in mind. $\endgroup$ – tchakravarty Dec 29 '13 at 20:00
  • 1
    $\begingroup$ @rocinante, please make contributions like this comments, rather than answers, or develop your points more thoroughly & don't include questions for clarification from the OP. W/ >50 reputation, you have the privilege to comment anywhere. $\endgroup$ – gung Dec 30 '13 at 4:54
3
$\begingroup$

The best way is to find out why there is a multicollinearity in your explanatory variables and remove it! Maybe looking at the pairwise correlation among your explanatory variables will give you an idea. In addition to this, there are some other ways for dealing with multicollinearity including:

  1. "Do nothing"! Believe it or not this is one simple way but most of the times it is not recommended.
  2. Dropping some of the variables is another solution, but it is like putting your head in the sand!
  3. Using an unweighted or weighted average of two variables that are highly correlated, but only if it seems logical. However, this only works when the two variables are actually a replications of one variable.
  4. Centering the variables by subtracting each explanatory variable from their means.
  5. You can use a little bit more advanced method to remove multicollinearity such as Principal Component Analysis. Here, you create a new set of variable (that are a linear combination of your explanatory variables) and use this new set of variables for your modeling. These new set of variables will be constructed in such a way that they are uncorrelated and therefore there will be no multicollinearity. In other words, you are re-defining your explanatory variables, by using some statistical analysis. See e.g. page 192 of Regression Analysis by Rudolf J. Freund, William J. Wilson.
  6. Sometimes, you have some prior knowledge about your explanatory variables and you will use this knowledge to re-define some new variables even without applying the Principal Component Analysis. For example, you may decide to use $X_i/X_1$ or $X_i-X_1$, of course if they have some meaning when modeling.
  7. If your explanatory variables are like polynomials, then you can use orthogonal polynomials (such as Legendre polynomials) as well to reduce multicollinearity.
$\endgroup$
1
$\begingroup$

You might be right to consider ridge regression; that's more a matter of how extremely multicollinear your variables are than how many you have. The process of choosing a ridge parameter can be made a little simpler up front by using Cule & De Iorio's (2012) method, though there's plenty of complexity behind the scenes there, and I haven't parsed it all yet myself, so take that recommendation with caution, if at all.

@Stat's recommendations are solid too, but deciding the best option will depend on what you know about your explanatory variables, if anything. If some of them are fundamentally similar, combining them to estimate fewer latent factors, which you could then use as predictor variables, might make sense. This would gloss over any important distinctions among those variables that share a common factor, however small those distinctions might be...so it would be wise to do what you can to ensure those distinctions aren't important before replacing them in the regression with a linear combination of them.

Reference

Cule, E., & De Iorio, M. (2012). A semi-automatic method to guide the choice of ridge parameter in ridge regression. arXiv:1205.0686 [stat.AP].

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.