I'm currently working on a regression problem where a subset of the predictor variables are zero-sum. By zero-sum I don't mean they all sum to zero, I simply mean that increasing one implies a decrease in one, all or some of the others. This fact makes these variables correlated, but it also means that there is a built-in opportunity cost when one variable has a very high value since all the other predictors must be correspondingly lower.

I was just curious to see if anyone has dealt with this kind of predictor variables before and, if so, have any suggestions on how to deal with these types of predictors, especially in terms of variable selection?

  • 2
    $\begingroup$ If "zero sum" does not mean zero sum, what distinguishes this from any question about (multi)collinearity, as the signs of correlations are arbitrary in that context? To the extent that a constant-sum constraint is approximated, it is more likely that one of those predictors will be redundant and is better omitted. Modern regression diagnostics should help. $\endgroup$
    – Nick Cox
    Commented Jun 4, 2013 at 19:04
  • $\begingroup$ Is the result of this perfect multicollinearity? What are your goals for this analysis? (Note that variable selection is a very difficult activity, & may often not be necessary.) $\endgroup$ Commented Jun 4, 2013 at 19:36

1 Answer 1


It sounds like what you have are clusters of high influence points where, due to the correlation structure of observed predictors, you may have very large effect sizes that are associated with very small differences in the predicted values in intermediate ranges, but could hypothetically produce ludicrous fitted values for outlying ranges of these predictors.

There are many reasons why we may not care to ameliorate this issue:

  1. If the interest is in inference, then by adequately adjusting for the desired stratification variables, we conserve the interpretation of the regression coefficient for our parameter of interest. That is, that "a unit difference in $X$ is associated with a $\beta$ difference in $Y$ holding $U_1$, $U_2$, ... fixed.

  2. Despite the possibility for generating large values in prediction, these variable ranges are implausible or downright silly extrapolations (such as testosterone levels in pregnant men). It is important with any prediction model to validate that model in independent data. If you are actually able to sample data from an independent validation dataset, then that's an issue of representativeness of the training sapmle.

  3. If used for prediction, the model achieves an alternative purpose effectively. For instance, with risk prediction for breast cancer, we may have horribly calibrated models for which the estimated risk in women does not reflect their actual event rates. However, if we dichotomize the risk prediction, we may identify a very high risk stratification for which screening therapies can be effectively recommended. This risk stratification does not require a model to be calibrated at all.

To test for these issues, it may be worthwhile to look at a histogram of leverage, influential points to determine whether specific observations are driving trends. I do not discard such points. Jackknife and bootstrapped standard errors can also be used to assess sensitivity to model based assumptions about the distribution of error terms. You can trim points based on their leverage/influence to obtain some outlier robust regression parameter estimates, but I never advocate doing this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.