4
$\begingroup$

Say you are dealing with a movie database that has movies and their genres. Genre is a categorical variable but each movie can belong to more than one genre.

For example, Movie A may be Comedy and Romance, Movie B is Romance and Action and Movie C is Comedy.

How would you encode these categories for analysis? Normally, dummy variable encoding involves a categorical variable where each observation belongs to only one of the possible values.

My idea is to add new variables for each of the different genres and then 1/0 code each movie for whether they belong to it or not. However, would this then be suitable for linear regression?

$\endgroup$
  • 1
    $\begingroup$ Does each movie have at least 1 genre? Is there a limit on the number of genres to which a single movie can belong, for example 3? $\endgroup$ – DeltaIV Dec 30 '17 at 12:16
  • 1
    $\begingroup$ Your idea seems good. You might however consider a model in which movies that belong to many categories have the effects of each category count less strong. For instance if the categories comedy, romance, drama have a certain similar (positive/negative) effect size than this should probably not be tripled for movies that happen to belong to all three categories.This is especially important if movies have been categorised with different 'intensity' of number of categories. $\endgroup$ – Sextus Empiricus Dec 30 '17 at 12:22
  • $\begingroup$ @DeltaIV unfortunately there is no limit to the number of genres, and since I believe this is user tagged data, some movies have quite a lot. $\endgroup$ – user190011 Dec 30 '17 at 15:39
  • 1
    $\begingroup$ @user190011 no problem, recommender systems can deal with this. How many genres do you have in total? $\endgroup$ – DeltaIV Dec 30 '17 at 18:34
0
$\begingroup$

Partially answered in comments, a summary:

You have a good idea, basically, but also other ideas can be tried.

You might however consider a model in which movies that belong to many categories have the effects of each category count less strong. For instance if the categories comedy, romance, drama have a certain similar (positive/negative) effect size than this should probably not be tripled for movies that happen to belong to all three categories.This is especially important if movies have been categorised with different 'intensity' of number of categories.

– Martijn Weterings

But how? Let the number of genres marked for a film be $k$. Then using a linear model with $$ \DeclareMathOperator{\E}{\mathbb{E}} \E Y=\mu =\beta_0+\beta_A x_A + \beta_B X_B + \dotsm $$ and $x_A, x_B, \dotsc$ being the indicators for genres $A,B,\dotsc$ and $k=x_A+x_B+\dotsm$ then we could replace the model above with $$ \E Y=\mu =\beta_0+\beta_A x_A/k + \beta_B X_B/k + \dotsm $$ maybe, that is, replace the 0,1 indicators with $0,1/k$ indicators. Maybe you could also include $k$ (or some function of it) as a covariable. You could also try to estimate the effect of $k$, but that could lead to a nonlinear model. I would try out various possibilities.

Then another advice was

No problem, recommender systems can deal with this. How many genres do you have in total?

– DeltaIV

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.