# Handling linear dependence among (non-mutually exclusive) binary predictors in linear regression

I have a number of movies tagged with varying genres.

|Title                 |Genres                                   |
|:---------------------|:----------------------------------------|
|10 Cloverfield Lane   |Drama, Horror, Mystery, Sci-Fi, Thriller |
|1408                  |Horror, Mystery                          |
|One Eight Seven       |Drama, Thriller                          |
|2 Days in the Valley  |Comedy, Crime, Thriller                  |
|2001: A Space Odyssey |Adventure, Sci-Fi                        |


I turned these into features for my linear regression model by creating a binary, indicator variable for each genre tag. These indicator variables aren't mutually exclusive in any way - a single movie can have multiple genre tags. For example, looking at Horror and Thriller tags (there are many more, 38 total tags/binary variables):

 > knitr::kable(head(data[ , c('Title', 'Genres', 'g_Thriller', 'g_Horror')]))

|Title                 |Genres                                   | g_Thriller| g_Horror|
|:---------------------|:----------------------------------------|----------:|--------:|
|10 Cloverfield Lane   |Drama, Horror, Mystery, Sci-Fi, Thriller |          1|        1|
|1408                  |Horror, Mystery                          |          0|        0|
|One Eight Seven       |Drama, Thriller                          |          1|        0|
|2 Days in the Valley  |Comedy, Crime, Thriller                  |          1|        0|
|2001: A Space Odyssey |Adventure, Sci-Fi                        |          0|        0|
|2012                  |Action, Adventure, Sci-Fi                |          0|        0|


My issue is that I have linear dependence among my genre binary variables:

> detect.lindep(data[ , genre_cols])
[1] "Suspiscious column number(s): 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 33, 34, 37, 38"
[1] "Suspiscious column name(s):   g_Drama, g_Horror, g_Comedy, g_Adventure, g_Action, g_Crime, g_Biography, g_Animation, g_Mystery, g_Short, g_Fantasy, g_Thriller, g_Western, g_Sci_Fi, g_Romance, g_Horror.1, g_Mystery.1, g_Thriller.1, g_Crime.1, g_Sci_Fi.1, g_Adventure.1, g_Drama.1, g_Fantasy.1, g_Biography.1, g_Comedy.1, g_Action.1, g_Western.1, g_Romance.1, g_Animation.1, g_Short.1"


For a case of mutually exclusive indicator variables (ie the dummy variable trap), I'm familiar that we leave one of the categories out to prevent linear dependence, but in the non-mutually exclusive case, I'm not sure how to handle. The volume of data and number of binary variables also makes it difficult to ascertain what specific combinations of my variables are linearly dependent. Are there ways to determine which of my non-mutually exclusive binary variables to drop, to remedy linear dependence? Or, other solutions to this linear dependence that I'm not seeing? I'm stuck using a linear regression model, so I can't change my modeling method.

My thinking was that I might look at the correlation of each of my genre variables against the response, and drop those with the lowest correlation until linear dependence was no longer an issue.

# Get correlation matrix for genres against response
> genre_cormat <- cor(data[ , c('y', genre_cols)])
> genre_cormat <- genre_cormat['y', ]
> knitr::kable(genre_cormat)
|              |     x|
|:-------------|-----:|
|y             |  1.00|
|g_Drama       |  0.02|
|g_Horror      | -0.03|
|g_Comedy      | -0.05|
|g_Action      |  0.01|
|g_Crime       | -0.09|
|g_Biography   |  0.06|
|g_Animation   |  0.03|
|g_Documentary |  0.06|
|g_Mystery     |  0.01|
|g_Short       |  0.03|
|g_Fantasy     | -0.05|
|g_Thriller    | -0.16|
|g_Western     |  0.00|
|g_Sci_Fi      | -0.17|
|g_Romance     | -0.02|
|g_Horror.1    | -0.03|
|g_Mystery.1   |  0.01|
|g_Thriller.1  | -0.16|
|g_Crime.1     | -0.09|
|g_Sci_Fi.1    | -0.17|
|g_Drama.1     |  0.02|
|g_Music       |  0.01|
|g_Fantasy.1   | -0.05|
|g_Biography.1 |  0.06|
|g_Comedy.1    | -0.05|
|g_Action.1    |  0.01|
|g_War         |  0.05|
|g_Family      |  0.04|
|g_Sport       | -0.02|
|g_Film_Noir   |  0.05|
|g_Western.1   |  0.00|
|g_Romance.1   | -0.02|
|g_History     |  0.00|
|g_Musical     | -0.05|
|g_Animation.1 |  0.03|
|g_Short.1     |  0.03|


So in the above correlation matrix against my response, I would first drop g_History as the lowest correlation 0.00012, recheck for linear dependence, and continue until remedied. But this seems crude - I'm not a fan of correlation between a binary and a numerical variables, and I'm not sure if this is the best way to approach in terms of predicting my model.

• You should start by looking at the correlations between the predictors. I expect there are some very high (say, 0.9) correlations among the genres. Consider merging these. If some genres are very rare (say, less than 5%), I would sooner delete them. That likely (but not sure) remove the collinearity issue. – ttnphns Aug 5 '19 at 20:51
• There are some suggestions for dealing with linearly dependencies among predictors on this page. I agree with @ttnphns (one contributor to that linked page) that focusing on correlations among predictors will be better than focusing on predictor correlations with response. – EdM Aug 5 '19 at 20:54
• Deleting predictors with very weak correlation with the predictand is not warranted, on the general grounds. Weak predictors could be informatively important because they still influence regressional effects of other predictors. More importantly, they may be valuable conceptually as they help make up the "field" or spectrum of genres. – ttnphns Aug 5 '19 at 20:56
• I edited the question, removing "dummy" and adding "binary". Non mutually exclusive binary "indicators" are not to call dummy. – ttnphns Aug 5 '19 at 21:09
• Have you consider using pca for dimensionality reduction? Is your goal to have a prediction model? What question are you trying to answer? – M Waz Aug 5 '19 at 21:25