0
$\begingroup$

I'm following a lecture that explains how to calculate item-item similarities using adjusted cosine distance (or Pearson correlation). I tried implementing this and have not gotten the same results.

This reproduces my experiment:

import pandas as pd

df = pd.read_json("./movie-ratings.json")

# This is how the lecture did it:
# Subtract the mean movie score from each movie column
item_mean_subtracted = df.sub(df.mean(axis=0), axis=1)

# Compute similarities with Pearson correlation
similar_item_matrix_1 = item_mean_subtracted.fillna(0).corr(method="pearson")

# This is how I think it should be done:
# Remove user rating bias by subtracting the mean user score from each user row.
user_mean_subtracted = df.sub(df.mean(axis=1), axis=0)

# Compute similarities again
similar_item_matrix_2 = user_mean_subtracted.fillna(0).corr(method="pearson")

Why does the lecture subtract the mean movie rating and not the mean user rating? I thought Also, I thought part of Pearson correlation was subtracting the mean. Why do I need to subtract the mean before doing Pearson correlation to get the numbers to look right?

$\endgroup$
1
$\begingroup$
  1. Why does the lecture subtract the mean movie rating and not the mean user rating?

There are two questions you could ask:

a) How much does Mike Miller like the "Lord of the Rings" relative to the other users?

b) How much does Mike Miller like the "Lord of the Rings" relative to the other movies he rated?

If you are interested in a), you would center using mean movie score. If you are interested in b), you would center using the mean user score. Therefore I conclude, that your lecturer is interested in a). Yet both are valid questions and both could potentially be input further processing.

  1. Why do I need to subtract the mean before doing Pearson correlation to get the numbers to look right?

Simply because that's what your lecturer did. Because your definition of "right" is "what my lecturer did". I assume he gave a great argument in the lecture, which we both missed. In general, both potentially are of (research) interest. Yet when building a recommender system, you are probably being taught a very specific way (features, classifier, training) of how someone did it. This does not mean, that it is the (only) right way.

Best way to go would be to use your input and his input for a recommender system and compare the results. .

$\endgroup$
  • $\begingroup$ I'm sorry but my definition of "right" isn't what my professor did...I'm trying to be objective here. Part of the definition of Pearson correlation is subtracting the mean. I'm curious why the lecturer did it before using Pearson correlation, since that implies to me that the mean is subtracted twice. I'm not saying he is "right". Thank you for your input though, this is really helpful. $\endgroup$ – The Puma Jan 14 '17 at 15:28
  • $\begingroup$ Implicitly - computing the covariance - the data is centered, yes. But before that, you are centering the data with a center which is not incorporated in the user data, but a movie related center. This is a different centering. $\endgroup$ – Nikolas Rieble Jan 14 '17 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.