1
$\begingroup$

I have a number of movies tagged with varying genres.

|Title                 |Genres                                   |
|:---------------------|:----------------------------------------|
|10 Cloverfield Lane   |Drama, Horror, Mystery, Sci-Fi, Thriller |
|1408                  |Horror, Mystery                          |
|One Eight Seven       |Drama, Thriller                          |
|2 Days in the Valley  |Comedy, Crime, Thriller                  |
|2001: A Space Odyssey |Adventure, Sci-Fi                        |
|2012                  |Action, Adventure, Sci-Fi                |

I turned these into features for my linear regression model by creating a binary, indicator variable for each genre tag. These indicator variables aren't mutually exclusive in any way - a single movie can have multiple genre tags. For example, looking at Horror and Thriller tags (there are many more, 38 total tags/binary variables):

 > knitr::kable(head(data[ , c('Title', 'Genres', 'g_Thriller', 'g_Horror')]))


    |Title                 |Genres                                   | g_Thriller| g_Horror|
    |:---------------------|:----------------------------------------|----------:|--------:|
    |10 Cloverfield Lane   |Drama, Horror, Mystery, Sci-Fi, Thriller |          1|        1|
    |1408                  |Horror, Mystery                          |          0|        0|
    |One Eight Seven       |Drama, Thriller                          |          1|        0|
    |2 Days in the Valley  |Comedy, Crime, Thriller                  |          1|        0|
    |2001: A Space Odyssey |Adventure, Sci-Fi                        |          0|        0|
    |2012                  |Action, Adventure, Sci-Fi                |          0|        0|

My issue is that I have linear dependence among my genre binary variables:

> detect.lindep(data[ , genre_cols])
[1] "Suspiscious column number(s): 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 33, 34, 37, 38"
[1] "Suspiscious column name(s):   g_Drama, g_Horror, g_Comedy, g_Adventure, g_Action, g_Crime, g_Biography, g_Animation, g_Mystery, g_Short, g_Fantasy, g_Thriller, g_Western, g_Sci_Fi, g_Romance, g_Horror.1, g_Mystery.1, g_Thriller.1, g_Crime.1, g_Sci_Fi.1, g_Adventure.1, g_Drama.1, g_Fantasy.1, g_Biography.1, g_Comedy.1, g_Action.1, g_Western.1, g_Romance.1, g_Animation.1, g_Short.1"

For a case of mutually exclusive indicator variables (ie the dummy variable trap), I'm familiar that we leave one of the categories out to prevent linear dependence, but in the non-mutually exclusive case, I'm not sure how to handle. The volume of data and number of binary variables also makes it difficult to ascertain what specific combinations of my variables are linearly dependent. Are there ways to determine which of my non-mutually exclusive binary variables to drop, to remedy linear dependence? Or, other solutions to this linear dependence that I'm not seeing? I'm stuck using a linear regression model, so I can't change my modeling method.

My thinking was that I might look at the correlation of each of my genre variables against the response, and drop those with the lowest correlation until linear dependence was no longer an issue. 

# Get correlation matrix for genres against response
> genre_cormat <- cor(data[ , c('y', genre_cols)])
> genre_cormat <- genre_cormat['y', ]
> knitr::kable(genre_cormat)
|              |     x|
|:-------------|-----:|
|y             |  1.00|
|g_Drama       |  0.02|
|g_Horror      | -0.03|
|g_Comedy      | -0.05|
|g_Adventure   | -0.17|
|g_Action      |  0.01|
|g_Crime       | -0.09|
|g_Biography   |  0.06|
|g_Animation   |  0.03|
|g_Documentary |  0.06|
|g_Mystery     |  0.01|
|g_Short       |  0.03|
|g_Fantasy     | -0.05|
|g_Thriller    | -0.16|
|g_Western     |  0.00|
|g_Sci_Fi      | -0.17|
|g_Romance     | -0.02|
|g_Horror.1    | -0.03|
|g_Mystery.1   |  0.01|
|g_Thriller.1  | -0.16|
|g_Crime.1     | -0.09|
|g_Sci_Fi.1    | -0.17|
|g_Adventure.1 | -0.17|
|g_Drama.1     |  0.02|
|g_Music       |  0.01|
|g_Fantasy.1   | -0.05|
|g_Biography.1 |  0.06|
|g_Comedy.1    | -0.05|
|g_Action.1    |  0.01|
|g_War         |  0.05|
|g_Family      |  0.04|
|g_Sport       | -0.02|
|g_Film_Noir   |  0.05|
|g_Western.1   |  0.00|
|g_Romance.1   | -0.02|
|g_History     |  0.00|
|g_Musical     | -0.05|
|g_Animation.1 |  0.03|
|g_Short.1     |  0.03|

So in the above correlation matrix against my response, I would first drop g_History as the lowest correlation 0.00012, recheck for linear dependence, and continue until remedied. But this seems crude - I'm not a fan of correlation between a binary and a numerical variables, and I'm not sure if this is the best way to approach in terms of predicting my model.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ You should start by looking at the correlations between the predictors. I expect there are some very high (say, 0.9) correlations among the genres. Consider merging these. If some genres are very rare (say, less than 5%), I would sooner delete them. That likely (but not sure) remove the collinearity issue. $\endgroup$
    – ttnphns
    Commented Aug 5, 2019 at 20:51
  • $\begingroup$ There are some suggestions for dealing with linearly dependencies among predictors on this page. I agree with @ttnphns (one contributor to that linked page) that focusing on correlations among predictors will be better than focusing on predictor correlations with response. $\endgroup$
    – EdM
    Commented Aug 5, 2019 at 20:54
  • 2
    $\begingroup$ Deleting predictors with very weak correlation with the predictand is not warranted, on the general grounds. Weak predictors could be informatively important because they still influence regressional effects of other predictors. More importantly, they may be valuable conceptually as they help make up the "field" or spectrum of genres. $\endgroup$
    – ttnphns
    Commented Aug 5, 2019 at 20:56
  • $\begingroup$ I edited the question, removing "dummy" and adding "binary". Non mutually exclusive binary "indicators" are not to call dummy. $\endgroup$
    – ttnphns
    Commented Aug 5, 2019 at 21:09
  • $\begingroup$ Have you consider using pca for dimensionality reduction? Is your goal to have a prediction model? What question are you trying to answer? $\endgroup$
    – M Waz
    Commented Aug 5, 2019 at 21:25

1 Answer 1

Reset to default
2
$\begingroup$

First, check how you have constructed your data table. The list of "suspicious column names" contains 15 pairs of names like {g_Drama, g_Drama.1} and {g_Horror, g_Horror.1}. Furthermore, within each of those pairs, the correlations with the outcome variables are identical, at least to the 2 decimal places that you display. That suggests that each pair might just represent 2 copies of the same data, or are otherwise linearly dependent. That none of the genres lacking a corresponding name with a ".1" suffix (Music, War, Film_Noir, etc.) showed up in your suspicious columns strengthens this hunch. Accidental introduction of linear dependence while setting up data for analysis can be all to easy, and is the first thing to look for in troubleshooting.

Failing that, examine correlations among the genres rather than their correlations with your response, as suggested by @ttnphns in a comment. As noted in the comments, removing predictors based on weak correlations with response can pose problems.

I'd start with pairwise correlations, looking for +1 or -1 correlations. There are several hundred of them (even if you limit to the 30 "suspicious" columns), but it would be easy to automate the search for such pairs.

If there isn't a simple explanation based on setting up the data table or pairwise correlations, then consult this page for several ways to test and correct for more complicated linear dependences.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.