I'm following a lecture that explains how to calculate item-item similarities using adjusted cosine distance (or Pearson correlation). I tried implementing this and have not gotten the same results.
This reproduces my experiment:
import pandas as pd df = pd.read_json("./movie-ratings.json") # This is how the lecture did it: # Subtract the mean movie score from each movie column item_mean_subtracted = df.sub(df.mean(axis=0), axis=1) # Compute similarities with Pearson correlation similar_item_matrix_1 = item_mean_subtracted.fillna(0).corr(method="pearson") # This is how I think it should be done: # Remove user rating bias by subtracting the mean user score from each user row. user_mean_subtracted = df.sub(df.mean(axis=1), axis=0) # Compute similarities again similar_item_matrix_2 = user_mean_subtracted.fillna(0).corr(method="pearson")
Why does the lecture subtract the mean movie rating and not the mean user rating? I thought Also, I thought part of Pearson correlation was subtracting the mean. Why do I need to subtract the mean before doing Pearson correlation to get the numbers to look right?