Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Today, I was playing around with a small dataset and performed a simple OLS regression which I expected to fail due to perfect multicollinearity. However, it didn't. This implies that my understanding of multicollinearity is wrong.

My question is: Where am I wrong?


I think that I can show that one of my variables is a linear combination of the others. This should lead to a regressor matrix that has no full rank and thus the coefficients should not be identified.

I generated a small reproducible dataset (code below):

   exporter importer      flow     dist intraUS
1    Canada   Canada  996.8677 6.367287       0
2   Florida   Canada  995.8219 9.190562       0
3     Texas   Canada 1001.6475 4.359063       0
4    Mexico   Canada 1002.4371 7.476649       0
5    Canada  Florida 1002.8789 5.389223       0
6   Florida  Florida 1007.5589 6.779686       1
7     Texas  Florida  996.8938 1.570600       1
8    Mexico  Florida 1005.6247 5.910133       0
9    Canada    Texas  999.9190 7.887672       0
10  Florida    Texas 1004.1061 7.187803       1
11    Texas    Texas 1004.5949 7.564273       1
12   Mexico    Texas 1000.3728 2.021297       0
13   Canada   Mexico 1003.0991 5.887743       0
14  Florida   Mexico  999.2210 3.058495       0
15    Texas   Mexico  997.6092 6.835883       0
16   Mexico   Mexico 1006.7934 5.794425       0

Each time exporter and importer are US states, the dummy intraUS is 1.

Now I perform a regression of (trade) flows on exporter and importer dummies, distance and the intraUS dummy. Feeding R with the following formula lm(flow ~ dist + exporter + importer + intraUS, data = dat) returns estimates for all coefficients, no missing values or warnings about singularity:

(Intercept)            dist exporterFlorida   exporterTexas  exporterMexico importerFlorida   importerTexas  importerMexico        intraUS1 
995.1033157       0.5744661      -1.2340338      -1.8792073       3.7375783       3.0361727       1.3256032       3.3225512       4.2429599

This puzzles me, because the regressor matrix clearly indicates that intraUS is a linear combination of exporterFlorida, importerFlorida, exporterTexas and importerTexas:

> mmat <- data.frame(model.matrix(lm(flow ~ dist + exporter + importer + intraUS, data = dat)))

   X.Intercept.     dist exporterFlorida exporterTexas exporterMexico importerFlorida importerTexas importerMexico intraUS1
1             1 6.367287               0             0              0               0             0              0        0
2             1 9.190562               1             0              0               0             0              0        0
3             1 4.359063               0             1              0               0             0              0        0
4             1 7.476649               0             0              1               0             0              0        0
5             1 5.389223               0             0              0               1             0              0        0
6             1 6.779686               1             0              0               1             0              0        1
7             1 1.570600               0             1              0               1             0              0        1
8             1 5.910133               0             0              1               1             0              0        0
9             1 7.887672               0             0              0               0             1              0        0
10            1 7.187803               1             0              0               0             1              0        1
11            1 7.564273               0             1              0               0             1              0        1
12            1 2.021297               0             0              1               0             1              0        0
13            1 5.887743               0             0              0               0             0              1        0
14            1 3.058495               1             0              0               0             0              1        0
15            1 6.835883               0             1              0               0             0              1        0
16            1 5.794425               0             0              1               0             0              1        0

Calculating exporterFlorida * importerFlorida + exporterFlorida * importerTexas + exporterTexas * importerFlorida + exporterTexas * importerTexas gives - not surprisingly - exactly the values in intraUS1.

So my question is, again: Why does this regression not fail, given that one variable is a linear combination of the others?


Below the complete code the reproduce the estimation:

## Generate data ####

set.seed(1)
states <- c("Canada", "Florida", "Texas", "Mexico")
dat <- expand.grid(states, states)
colnames(dat) <- c("exporter", "importer")

dat[, "flow"] <- NA
dat[, "dist"] <- NA
dat[, "intraUS"] <- 0

for (i in 1:nrow(dat)) {
  dat[i, c("flow", "dist")] <- c(rnorm(1, mean = 1000, sd = 5), rnorm(1, mean = 6, sd = 2))
  if (dat[i, "exporter"] %in% states[2:3] && dat[i, "importer"] %in% states[2:3]) {
    dat[i, "intraUS"] <- 1
  }
}
dat$intraUS <- factor(dat$intraUS)

## Run regression - works! ####

summary(lm(flow ~ dist + exporter + importer + intraUS, data = dat))

## Show that "intraUS1" is a linear combination of the dummies. ####

mmat <- data.frame(model.matrix(lm(flow ~ dist + exporter + importer + intraUS, data = dat)))

cbind(mmat, test = with(mmat,
                        exporterFlorida * importerFlorida + exporterFlorida * importerTexas +
                        exporterTexas * importerFlorida + exporterTexas * importerTexas
                        ))[, c("intraUS1", "test")]
share|improve this question
8  
Downvoters, would you mind explaining what I did wrong? I'm not so familiar with Crossvalidated and would like to know what's wrong with this question, so I can do better the next time. – CL. Feb 9 at 16:11
up vote 27 down vote accepted

exporterFlorida * importerFlorida + exporterFlorida * importerTexas + exporterTexas * importerFlorida + exporterTexas * importerTexas

This is not a linear combination of exporterFlorida, importerFlorida, importerTexas and exporterTexas. In a linear combination, the coefficients of the vectors must be constants. So something like

2*importerFlorida + 3*importerTexas - exporterFlorida - 2*exporterTexas

is a linear combination.

What you have could possibly be called a quadratic combination, but that's stretching terminology into "I'm making stuff up" land.

share|improve this answer
1  
Damn … this is the punishment for never having attended a math course but starting with statistics instead. Thank you! – CL. Feb 9 at 16:00
3  
No worries. That's what we're here for! – Matthew Drury Feb 9 at 16:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.