Can I use all pairwise comparisons at a factor level to perform correlation analysis?

Question

I have 4 data sets, each with thousands of records/instances. There are 2 factors. One factor has 3 instances at each factor level (there are thousands of levels in the data set). The other factor also has thousands of levels, but with varying numbers of instances at each level. For each factor, I would like to measure how similar the values are. The variable whose similarity I want to measure is non-Normally distributed in at least 1 of the data sets.

One idea was to create a table with all pairwise comparisons at each factor level and do correlation analysis. i.e., for the first factor, I would get a table that looks like this:

factor_level col1 col2
factor1 val1  val2
factor1 val1  val3
factor1 val2  val3
factor2 val1  val2
factor2 val1  val3
factor2 val2  val3
... 

and that would be used to calculate correlation coefficients (col1 vs. col2). For the first factor, I would like to get one correlation coefficient for the entire data set (to compare to another published data set for which I would perform the same analysis).

The table for the second factor looks like this:

factor_level col1 col2
factor1 val1 val2
factor1 val1 val3
factor1 val1 val4
factor1 val2 val3
factor1 val3 val4
factor2 val1  val2
factor2 val1  val3
factor2 val2  val3
...

For the second factor, I would also like to calculate the correlation at each factor level (in the end, I would like to rank the factor levels in my data set by the similarity of their values and design some experiments to test the highest-ranking ones).

After doing this, I got results that seemed to make sense (i.e. the correlations, Spearman's rho, were high in cases where I would expect that based on published data and low where I would expect them to be low).

Out of curiosity, I decided to try another method to calculate correlation coefficients. I randomly split the data at each factor level into 2 groups (if there were 3 rows at a factor level, I left one of them out) and paired up 1 row from each group. The table I used to do the correlation analysis looks like this:

factor_level group1_val group2_val
factor1 val1  val1
factor2 val1  val1
factor2 val2  val2
factor2 val3  val3
...

I repeated this analysis 10 times. That is, I did the random splitting into 2 groups 10 times and calculated the mean Spearman's rho across the 10 runs.

The correlation coefficients using this method were generally lower than what I obtained using the all-pairwise comparisons. I'm sure there is an obvious explanation for this. In a way, I am not disturbed by the differences, because the trends are still the same (i.e. those 'control' cases where I expect high or low correlations are still ranked as expected). And the rankings based on correlation coefficients at the second factor level are similar. My question, though, is whether it is considered appropriate to calculate correlation coefficients as I have done (i.e. using all pairwise comparisons). Or whether it's something other people have done. I don't have 2 variables, so it seems wrong to do it as I have. I'm guessing it might be better to use some form of ANOVA, but I would be happy to use my correlation coefficients so long as their use is not too egregious.

Thank you in advance for your help.

Answer 1

Seems that what I was looking for is the intraclass correlation.

edit:

In statistics, the intraclass correlation (or the intraclass correlation coefficient, abbreviated ICC)1 is a descriptive statistic that can be used when quantitative measurements are made on units that are organized into groups. It describes how strongly units in the same group resemble each other. While it is viewed as a type of correlation, unlike most other correlation measures it operates on data structured as groups, rather than data structured as paired observations.

The groups in the above description are the factor levels from the question. Instead of calculating a correlation coefficient based on pairing observations at each factor level (either all pairwise or after randomly splitting the data set in 2), I calculated the ICC (did so using the R ICC package).