2
$\begingroup$

If you have two strongly correlated variables, lets say Ice Cream Sales (ICS) and Temperature (T). If you ran 2 linear regressions, one with T as the predictor, the other with ICS as the predictor - with Shark Attack Rates (SAR) being the target variable - both would likely reveal a relationship. However, we would probably argue that any variance explained by ICS is really explained by T.

However, if we ran a multiple linear regression passing both ICS and T in, how would the algorithm determine what the true predictor is? Would it be able to determine that ICS only shares a relationship with SAR because it correlates with T? Or is it just up to the analyst to make judgements about what predictors to include in the model - in these case, removing ICS?

$\endgroup$
3
$\begingroup$

I think it's easier to understand how regression "magically" figures out which variable is the "real" predictor using binary predictor variables instead of continuous ones.

We have a dataset of different days. Each day can be either hot or cold (T), and ice cream sales (ICS) on that day can be either high or low. Also there is a certain "SAR" score on that day. Looking at each independent variable individually we see higher SAR scores on BOTH hot days (compared to cold days) and on high ICS days compared to low ICS days. How can we figure our whether T or ICS is really driving SAR?

Well, let's start by ONLY looking at "high" ICS days. We find that, just among high ICS days there were 24 attacks on the hot high ICS days, but only 10 on the cold high ICS days. Now let's look only at low ICS days - we find that among days with low ice cream sales there were 12 attacks on the hot days but only 4 on the cold days.

If the relationship between T and SAR were REALLY due to high ICS, then we wouldn't still see that relationship when we limit our analysis to high (or low) ICS days. but we DO still see it, which suggests that ICS is NOT driving the relationship between T and SAR. We've "controlled" for ICS by running our T analysis separately for different values of ICS, Let's do the same thing for ICS, controlling for temp,

Looking only at "hot" days we see that there were 25 attacks on high ICS hot days and 24 on low hot ICS days. Looking only at "cold" days we see that there were 2 attacks on high ICS cold days and 3 on low ICS cold days. Here the relationship between ICS and attacks that we saw before has disappeared when we analyze the data separately for hot and cold days. This suggests that the relationship we saw before WAS merely due to the fact that hot days are more likely to have high ICS.

This is a simplified version of what multiple regression is doing - it's looking at the relationship between a given independent variable and the dependent variable holding all of the other variables "constant" in the way we just did with T and ICS. The difference is that it can do this with multiple variables simultaneously, and it accounts for all of the possible values that continuous variables can take on - so it can account for the fact that there are more values of temp aside from just "hot" and "cold."

So if you actually ran a linear regression model of SAR as a function of both T and ICS (treated now as continuous variables) the coefficient for T tells you how the model things SAR will change if you increase T by one unit (degree?) but "hold ICS constant" while the ICS coefficient tells you what happens to ISAR if you increase ICS by one unit and hold Temp constant.

Since the relationship is (we assume) really due to temperature and not ice cream, we expect that if you had two different days with the same ice cream sales but different temperatures, you'd expect more attacks on the hotter day, and that's what a significant (positive) temp coefficient would tell you.

$\endgroup$
5
  • $\begingroup$ Thank you for this! Perfectly explained. I was struggling to understand while reading about multiple LR, but this helped me to grasp it. $\endgroup$ Mar 12 at 11:22
  • $\begingroup$ And another point, it could be possible that every hot day has high ice cream sales, and every cold day has low ice cream sales - and every day with high ice cream sales is a hot day, and every day with low ice cream sales is a cold day. Not likely, but in this scenario, would the model simply tell you that both variables explain the variance - and not be able to identify that ice cream sales isn't a real predictor? $\endgroup$ Mar 12 at 11:39
  • $\begingroup$ Aaron Wright - what you are describing is a well known problem called "multicolinearity." There does come a point where two variables are SO closely correlated that the model can't distinguish one from the other. When that happens the model either doesn't run at all or gives you bogus/bad/nonsensical answers. If T and ICS are both measured as continuous, then this is less likely to occur, but this general problem is something we worry about a lot. $\endgroup$ Mar 12 at 13:01
  • $\begingroup$ Ah I see. Thanks Graham. I'll do some reading on mutlicolinearity. $\endgroup$ Mar 12 at 13:09
  • $\begingroup$ Nice explanation (+1) for a scenario in which one predictor is at root responsible for virtually all of the explanation. For more complex or nuanced situations, one certainly does need to make extra-statistical judgments about which variables should be treated as worth including or worth being attributed with ("credited with") shared variance that could be attributed elsewhere. $\endgroup$
    – rolando2
    Mar 14 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.