0
$\begingroup$

As part of my project, I've been trying to analyse (and hopefully make some knowledgeable conclusions about) the movie database dataset, which consists of the following columns:

  1. Movie ID - ID of a particular movie, for example: 123123
  2. Subtitles - list of the subtitles available for said movie from the set of [English, French, Spanish, Chinese]
  3. Maturity - maturity level assigned to the movie from the set [3+, 7+, 12+, 18+]
  4. Genres - list of genres, for example [comedy, romantic, european]
  5. Runtime - length of a movie in minutes, for example: 90
  6. Directors - list of directors responsible for movie, for example [Anna Migotto, Sabina Fedeli]
  7. Actors - List of major actors, which played in said movie, for example [Helen Mirren, Gen Ghergatti]
  8. Synopsis - short description of the movie plot, for example: 'John Keating, a progressive English teacher, tries to encourage his students to break free from the norm, go against the status quo and live life unapologetically.'

My aim is to be able to cluster the movies based on those parameters, to both:

  1. "Reduce dimensionality" - a bit wrongly stated, but the idea is to be able to use said clusters to reduce the number of data used in further analysis of vierwership. As of now dataset even after some work containes more than 200 unique genres, not to mention actors and/or directors.
  2. Analyse and discover relations in the dataset - the underlying knowledge about the relations between the movies and how the recommendations could be made based on this is strongly needed.

As of now I was able to do the following:

  1. Clear the data: there are no NA's in the dataset (or at least no in any "major" area, like genres and/or synopsis). Also synopsis has been both lowercased, lemmatized and cleared from any stopwords and is now a list of tokens. Moreover, subtitles and genres have been made lowercased. In case of genres, they have been uniformized (ie, there are no comedy and comedies as separate genres).
  2. Get descriptive statistics - word counts, unique word counts, most common words, number of genres etc.
  3. Make TF-IDF analysis with PCA/K-means - that one is a bit tricky. My first attempt to the aforementioned problem was to make a "soup" from the above data, after some "grouping" (in that case, making groups for runtime, like <1h, 1-1.5h etc.) and treating it to some old fashioned PCA/K-Means clustering after TF-IDF vectorization. I know that this approach is wildly inappropriate, as it puts EVERYTHING in one bag and treats it equally, but hey! We've got to start somewhere. Nevertheless clusters got by this were quite useful so far. From technical side this has been coded in Python, using sklearn, pandas and to some extent sparse matrices.

What is my aim:

  1. Make a proper analysis - in that regard I have read about Gowers distance, as well as PAM (https://www.kaggle.com/flubber/clustering-gower-distance-pam, https://medium.com/analytics-vidhya/gowers-distance-899f9c4bd553, https://www.cs.umb.edu/cs738/pam1.pdf) algorithm and both of them seem to fit my problem perfectly, not to mention that they shouldn't put everything to "one bag" as it happens now.

What is my problem: I don't know if my idea and approach to the problem are appropriate as of now.

The Question: Is this approach valid? What I mean by that is event he idea of describing movies in such a way for my purposes right, or am I go astray with those algorithms?

I will appreciate any help, starting with links to some sklearn tutorials, and ending with criticism of my approach. If any data/code example is needed I will post it on short notice. Thank you in advance!

This is the followup question to: Clustering mixed data based on text anlysis: Sparse Matrix problem

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ You can define different distances and aggregate them using Gower's scheme, and then cluster with PAM, that seems reasonable to me. I don't quite know what the TF-IDF/PCA/K-means bit is doing there. As far as I'm concerned, you could go into Gower/PAM without doing some information reduction before. The key thing is to define appropriate distances for each of the variables, most tricky probably for Synopsis, and maybe variable weighting. $\endgroup$
    – Christian Hennig
    Oct 22, 2020 at 9:15
  • $\begingroup$ TF-IDF is just for the later part you have mentioned. I need a way to quantify "text" data somehow, and TF-IDF helps me accomplish just that, ie. creates a numerical representation of text, which can be analyzed. But the dimensionality of such representation can be quite bothersome (~100k if not restrained properly), so the PCA helps with that a lot. K-means on the other hand would be quite good if we have dealt with only numerical, not categorical variables, so in that regard it can be quite misleading here. $\endgroup$
    – Andrzej Chomiczewski
    Oct 22, 2020 at 9:48
  • $\begingroup$ If this is for the synopsis data, fair enough, I don't know any better. For directors and actors I'd imagine one can generate a straight distance based on overlap. $\endgroup$
    – Christian Hennig
    Oct 22, 2020 at 10:41
  • $\begingroup$ Straight distance? Could you elaborate? When I thought about it, it seemed to me as those factors can generate very wast distances due to the fact, that in the dataset we don't often see reoucurring names in the crew, outside of some major names. $\endgroup$
    – Andrzej Chomiczewski
    Oct 22, 2020 at 14:31
  • $\begingroup$ "Straight" is not a name or technical term. One could choose a function of "number of people in common", like max number of people in common minus number of people in common, and then if all distances are too big maybe transform this in a nonlinear manner. $\endgroup$
    – Christian Hennig
    Oct 22, 2020 at 15:10

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.