Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In a simple linear regression model with two independent variables, if there is a strong correlation found between the variables, it is suggested that we should include only one of them in the model.

I am building a model where the dependent variable is Lunch Cost and the independent variables are Number of Class 1 Students Buying Lunch and Number of Class 2 Students Buying Lunch. However, I found there was a strong correlation between the two independent variables; when Number of Class 1 Students Buying Lunch increases, Number of Class 2 Students Buying Lunch increases as well.

In this case, removing one of them from the model seems not ideal since my goal is to know how the two independent variables would describe Lunch Cost. What should I do here? Thanks!

share|improve this question
up vote 5 down vote accepted

The first question to ask is: do you actually need to care? If you're just trying to predict the cost of future lunches, then this isn't really an issue. On the other hand, if you're trying to assess the relative contributions of Class #1 and Class #2 students to the cost, then collinearity is a bigger problem.

In a well-behaved, non-colinear model, we might take a model like $y = \beta_0 + \beta_1 \cdot x_1 + \beta_2 \cdot x_2$ and fit it with our data to find the $\beta$ values. We might find that $\beta_1 = 2$ and $\beta_2 = -0.5$, which would indicate that a one unit increase in $x_1$ results in a 2 unit increase in $y$, while a similar change in $x_2$ causes a half-unit decrease in $y$. However, if $x_1$ and $x_2$ are highly correlated, this interpretation goes right out the window.

Suppose we fit a model $Y = \beta_0 + \beta_1 \cdot x_1$ and found that $\beta_0 = 0$ and $\beta_1 = 4.$ Everything's great! Now we do something dumb and fit this model instead $Y = \beta_0 + \beta_1 \cdot x_1 + \beta_2 \cdot x_2$, where $x_1 = x_2$ (in other words, $x_1$ and $x_2$ are completely correlated).

In this case, we can pick literally any set of $\{\beta_1, \beta_2\}$ values that add up to four: (2,2), (1,3), (1003, -999), and so on: these are all the points on the line $x+y=4$ (hence the name!). These all give you the same prediction, but depending on your choice you would be claiming that a 1 unit increase in $x_1$ is associated with a 2, 1, or 1003 unit increase in $y$, respectively, which can't all be correct! This is obviously an extreme example, but you could imagine similar things happening when the $x_s$ are somewhat less strongly correlated.

I'm also tempted to ask why you're separating out students by class--is there some reason to think that Class #1 and Class #2 students contribute differently to the price of lunch? Perhaps a model where you regress lunch cost ~ total number of students would be more appropriate?

share|improve this answer
The main reason to think that the two classes would contribute differently to the price of lunch is that Class 2 is older than Class 1 on average. I just want to predict the future lunch cost; however, should I mention the correlation that I found between the two classes in my report? – Guess Gucci Feb 20 '13 at 20:22
That seems pretty reasonable: "The model predicts [this and that], but due to multi-collinearity, we are unable to assess the relative contributions of Class 1 and 2." All that said, there are some ways of getting around it:… – Matt Krause Feb 20 '13 at 20:35
Thank you very much for your suggestions. – Guess Gucci Feb 20 '13 at 20:50
By the way, what do you mean by relative contribution? For instance, I have Y = 2*X_1 + 4*X_2 and am able to conclude that 1 unit increase in X_1 would lead to a 2 unit increase in Y. Is this the relative contribution you are talking about? Thanks again. – Guess Gucci Feb 20 '13 at 21:06
@GuessGucci: "The main reason to think that the two classes would contribute differently to the price of lunch is that Class 2 is older than Class 1 on average" then wouldn't separating this into total number of students (= class 1 + class 2) and difference between class 1 and class 2 be a sensible transformation? Which of course gets you on the way to re-invent principal component regression... – cbeleites Feb 20 '13 at 21:07

Based on the fact that it's the average age of Class 2 vs. Class 1 that (you hypothesize) may matter, you could try a model where the response is Lunch Cost, and the predictors are

  1. a factor for whether a student is in class 1 or class 2
  2. the student's age

This way, you can ask whether age matters, and whether belonging to class 2 (rather than class 1, which would be a baseline) also matters.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.