# How to deal with information of variable length?

This is the question arised from my previous question. Basically, I am trying to build a prediction model for movie's rating. So I have to deal with numeric attribute such as actor_id, director_id, budget, etc., in my data set. So my question is how I should deal with those attributes for my classification.

• Variables such as actor_id are not numeric in the sense of quantifying anything that is important to your analysis. I suppose they quantify the order in which the actors got listed, but that's all. So you should conceptualize each variable like that as a nominal (a.k.a. categorical) variable. <br><br> Beyond that, in order to obtain useful help, please say what type of procedure/analysis you are using and in what software. Next, please tell what you have tried and what exactly is getting you stuck. – rolando2 Nov 25 '11 at 1:54
• @KHein since you are a beginner in the area of data mining, you may find this thread helpful: stats.stackexchange.com/questions/18973/… – steffen Nov 26 '11 at 12:04
• I added the recommender-system tag. Feel free to correct this if I'm wrong. – chl Nov 26 '11 at 12:28

As rolando2 already pointed out, that also the actor_id is a number, it does not represent a numerical variable. The selection of the correct scale of a variable depends on its meaning. See this wikipedia page for an overview of scales.

By checking the IMDB-site I got a rough idea what your data may looks like. Roughly said, your data contains of two parts:

• fixed features with exactly one value for each movie (like: Category, Budget etc.).
• varying number of "information-pieces". This includes actors, directors (happens), Genre (IMDB has multiple for one film ... like a highlevel tag), Plot keywords etc. etc.

Strategies to model these "information-pieces"

Boolean Feature

One boolean feature of each value of a field, where the value indicates whether the field is active for the current movie or not. In the actor example this means that you will have a feature for every actor who has ever appeared in a movie (given your data), where "true" means that the actor appeared in the particular movie, "false" not.

Where this approach does not loses information by summarization, it has cleary the drawback that the number of features will implode.

I do not recommend this approach as long as you are not planning to model your data with graphs (but this is advanced stuff).

Text Mining has to deal with the same issue: Here the information-pieces are words (or tags) and the label is (in general) the category. I am pretty sure you can grab some ideas by checking the work in this area.

Numeric Feature (Summarization)

Instead of given every value a single feature and calculating the impact on this value of the rating, you just calculate the expected rating for all the values of this particular field (in advance) and create a single feature for it.

Example feature actors: Tom Cruise and Cameron Diaz played in Movie "Knight and Day". Now you calculate the average rating of the movies with Tom Cruise, the average rating the movies with Cameron Diaz and then calculate e.g. the average of both values weighted by the number of movies both appeared in respectively. This will be the value for the field "actors" for the Movie "Knight and Day".

The advantage of this method is a small overall number of features. The drawback is that too much summarization can harm the quality of the resulting model. You may want to combine this strategy with the next one (only using the most important actors of a movie, not all).

Nominal Feature (Truncation) Instead of using every value which has appears on a particular movie site, why not restrict the data to the most important ones. You can create features like

• first_actor,second_actor, ...
• director (one field)
• genre (one field)
• first_keyword, etc..

and use simply the value as it is (actor_id in case of first_actor, etc...). In this way the number of features wont explode and you have a good point to start.

A clear drawback explained in the actor case: In a Jim Carey Movie, the first actor is important, the second and third ... less. In Ocean's Eleven, the first (do not know, 11 ?) actors are important.

I am pretty sure by working with the data (i.e. putting the unstructured information from IMDB into tabular format) you will have more (domain specific) ideas. The most important part of Data Mining is to prepare the data the right way, so that a solid model only has to "read" the information. After you have find a good data representation, you can deal with finding the best algorithm or algorithm parameters to get from solid/good to a very good model quality.

• Thank you very much for such a great answer Steffen. I think I will try to limit to first 5 actor/actress first and see how the classification go. – K Hein Nov 27 '11 at 8:30