Measuring correlation for variables calculated from pairs of observations

Question

Say I have data for the music and film tastes of a set of users $U$ (let's say we have $|U| = 100$ users). Given two users $u_1$ and $u_2$, say I also have two functions $\mathrm{sim}_m(u_1,u_2)$ and $\mathrm{sim}_f(u_1,u_2)$ that give me similarity scores for the music and film tastes, respectively, for that pair of users. (If important, one can assume that the similarity scores yield values in $[0,1]$, that $\mathrm{sim}_*(u,u) = 1$, and that $\mathrm{sim}_*(u_1,u_2) = \mathrm{sim}_*(u_2,u_1)$, for example.)

Note that I cannot define a function to give an ordinal score to the music or film tastes of a particular user, but I do have such a function to give an ordinal score to the similarity of the music or film tastes of a pair of users.

I would like to address the question: is there a statistical relation between the similarities of the music and film tastes of users?

A first idea would be, for all pairs of users $(u_1,u_2) \in U \times U$, compute their music similarity and film similarity scores in a table with $|U|^2$ rows, and then apply a standard correlation measure between the two similarities. (An alternative included here would be to compute the table with $(|U|^2 - |U|) / 2 $ rows containing non-reflexive, non-symmetric pairs.)

A second idea would be, for each user $u \in U$, compute their music and film preferences to all other users $u' \in U \setminus \{u\}$, measure the correlation between the music and film preference for that user $u$ to all other users, and then present a distribution of correlation values across all users.

(There may be other ways to explore this statistical relation.)

The benefit of the first approach is that we get one correlation measure and associated $p$-value, but it's not clear to me if the $p$-value in particular is appropriate as the rows are not completely "independent" (each user influences at least $|U| - 1$ rows of the data). Taking the product (pairs) of users seems to superficially inflate the sample size, for example.

The benefit of the second approach is that the correlation is computed on "independent" data (each user is "counted" once in each correlation computed), but interpretation is more difficult as we do not have a single correlation measure nor $p$-value but rather a multiset of them.

---

Questions:

As the first approach provides the most direct interpretation, it might be preferible, but is it appropriate?

Otherwise, is there another (better) approach than the second one mentioned above to test for a stastical relationship between similarities of the music and film tastes of users?

Answer 1

The pairwise correlations over the non-symmetric, non-reflective case indeed allows a reasonably natural interpretation, but you are right that due to the dependencies, the resulting inferential statistics (p-values, confidence intervals) are invalid. Some ways to workaround this are:

Permutation test

A simple and direct way to obtain valid p-value is to do a permutation test. I.e. you compute the correlation (or any other statistics you want) from your actual data. Than you repeatedly randomly permute the pairing between the music and film tastes and for each of those random permutations you compute the same statistic. If the statistic for your actual data is in the tail of the distribution of the randomly permuted values, you can reject the null hypothesis of no relation. This process is related to PERMANOVA

A problem with this approach is that the null hypothesis of complete lack of correlation between the two tastes is pretty strong so you are almost guaranteed to reject it, even if the actual correlation is very low. (e.g. assume that the actual tastes for genres/style are independent, but french speaking people have some preference for french music AND french movies compared to english speaking people, than the null doesn't hold, but for possibly uninteresting reason).

Ordination and regression

Another way to approach the problem would be to do some type of ordination, like NMDS - this way you obtain a low-dimensional representation of your data (i.e. music taste for each person) such that the euclidean distance in this representation is a good proxy for the actual (dis)similarity of the tastes.

Then you can see how well you can predict the ordination results for film taste from ordination results for music taste (or vice versa) using standard statistical techniques (e.g. multivariate regression).

Comparing separate and joint ordination

Alternatively, you can try to do ordination of a combined distance for film and music and compare it to separate ordinations for film and music. E.g. if you need 3 dimensions to provide "good" represenation of the musical taste dissimilarities and 3 dimensions to provide "good" representation of the movie taste dissimilarities but you can get a similarly "good" representation of the total dissimilarity in 4 dimensions (for a suitable notion of "good"), it indicates that some information is shared between the two dissimilarities.