Background
Hi all, I need some clarification on my approach if it's correct or not. I have a matrix (M_ij) with user ratings of images. The users (i) are on the horizontal axis and the images (j) are on the vertical axis. The cell ij has the ratings given by the user. These ratings are on a discrete scale from (1-5). This is a sparse matrix since each user did not rate all the images but rather the subset of images.
My Approach
I found a correlation matrix of users. Since I had 756 users, the resulting correlation matrix had the dimensions 756 x 756. To cluster these users based on how similar their ratings were, I converted this matrix to a dissimilarity matrix doing $2 \times \sqrt{1 - s}$ to get $d$, the correlation distance (based on https://stats.stackexchange.com/a/36158). Then I used the ward's method by using $d$ as the distance metric to perform hierarchical clustering.
My Unclarities:
- While the above approach seems valid, I do not understand why a correlation matrix must be converted to a distance matrix for clustering. I read this post, but I could not understand that if correlation coefficients can be treated as points, then why go through the trouble of converting it into a distance matrix for clustering?
- What could be the reasons why this approach may be wrong?
- Should I have not used the Ward method and had used any other method for hierarchical clustering using $d$?
- Based on the geometrical properties (as mentioned here) of a positive semi-definite matrix, it must not have any negative eigenvalues. However, in my case, the correlation matrix had several negative eigenvalues, but yet I calculated $d$ from it. Is it wrong? or the resulting $d$ matrix was partially valid?
Thank you in advance :)
