In a simple linear regression model with two independent variables, if there is a strong correlation found between the variables, it is suggested that we should include only one of them in the model.

I am building a model where the dependent variable is Lunch Cost and the independent variables are Number of Class 1 Students Buying Lunch and Number of Class 2 Students Buying Lunch. However, I found there was a strong correlation between the two independent variables; when Number of Class 1 Students Buying Lunch increases, Number of Class 2 Students Buying Lunch increases as well.

In this case, removing one of them from the model seems not ideal since my goal is to know how the two independent variables would describe Lunch Cost. What should I do here? Thanks!


The first question to ask is: do you actually need to care? If you're just trying to predict the cost of future lunches, then this isn't really an issue. On the other hand, if you're trying to assess the relative contributions of Class #1 and Class #2 students to the cost, then collinearity is a bigger problem.

In a well-behaved, non-colinear model, we might take a model like $y = \beta_0 + \beta_1 \cdot x_1 + \beta_2 \cdot x_2$ and fit it with our data to find the $\beta$ values. We might find that $\beta_1 = 2$ and $\beta_2 = -0.5$, which would indicate that a one unit increase in $x_1$ results in a 2 unit increase in $y$, while a similar change in $x_2$ causes a half-unit decrease in $y$. However, if $x_1$ and $x_2$ are highly correlated, this interpretation goes right out the window.

Suppose we fit a model $Y = \beta_0 + \beta_1 \cdot x_1$ and found that $\beta_0 = 0$ and $\beta_1 = 4.$ Everything's great! Now we do something dumb and fit this model instead $Y = \beta_0 + \beta_1 \cdot x_1 + \beta_2 \cdot x_2$, where $x_1 = x_2$ (in other words, $x_1$ and $x_2$ are completely correlated).

In this case, we can pick literally any set of $\{\beta_1, \beta_2\}$ values that add up to four: (2,2), (1,3), (1003, -999), and so on: these are all the points on the line $x+y=4$ (hence the name!). These all give you the same prediction, but depending on your choice you would be claiming that a 1 unit increase in $x_1$ is associated with a 2, 1, or 1003 unit increase in $y$, respectively, which can't all be correct! This is obviously an extreme example, but you could imagine similar things happening when the $x_s$ are somewhat less strongly correlated.

I'm also tempted to ask why you're separating out students by class--is there some reason to think that Class #1 and Class #2 students contribute differently to the price of lunch? Perhaps a model where you regress lunch cost ~ total number of students would be more appropriate?

  • $\begingroup$ The main reason to think that the two classes would contribute differently to the price of lunch is that Class 2 is older than Class 1 on average. I just want to predict the future lunch cost; however, should I mention the correlation that I found between the two classes in my report? $\endgroup$ Feb 20 '13 at 20:22
  • $\begingroup$ That seems pretty reasonable: "The model predicts [this and that], but due to multi-collinearity, we are unable to assess the relative contributions of Class 1 and 2." All that said, there are some ways of getting around it: online.stat.psu.edu/online/development/stat501/… $\endgroup$ Feb 20 '13 at 20:35
  • $\begingroup$ By the way, what do you mean by relative contribution? For instance, I have Y = 2*X_1 + 4*X_2 and am able to conclude that 1 unit increase in X_1 would lead to a 2 unit increase in Y. Is this the relative contribution you are talking about? Thanks again. $\endgroup$ Feb 20 '13 at 21:06
  • 2
    $\begingroup$ @GuessGucci: "The main reason to think that the two classes would contribute differently to the price of lunch is that Class 2 is older than Class 1 on average" then wouldn't separating this into total number of students (= class 1 + class 2) and difference between class 1 and class 2 be a sensible transformation? Which of course gets you on the way to re-invent principal component regression... $\endgroup$ Feb 20 '13 at 21:07
  • $\begingroup$ GuessGucci, exactly. I've edited in an example, which I hope clears things up. One way to solve colinearity problems is to transform your data in a way that reduces the correlations between variables while keeping the variables interpretable. @cbeleites's solution is a great idea--you should give it a shot! $\endgroup$ Feb 21 '13 at 6:18

Based on the fact that it's the average age of Class 2 vs. Class 1 that (you hypothesize) may matter, you could try a model where the response is Lunch Cost, and the predictors are

  1. a factor for whether a student is in class 1 or class 2
  2. the student's age

This way, you can ask whether age matters, and whether belonging to class 2 (rather than class 1, which would be a baseline) also matters.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.