I have 4 data sets, each with thousands of records/instances. There are 2 factors. One factor has 3 instances at each factor level (there are thousands of levels in the data set). The other factor also has thousands of levels, but with varying numbers of instances at each level. For each factor, I would like to measure how similar the values are. The variable whose similarity I want to measure is non-Normally distributed in at least 1 of the data sets.
One idea was to create a table with all pairwise comparisons at each factor level and do correlation analysis. i.e., for the first factor, I would get a table that looks like this:
factor_level col1 col2 factor1 val1 val2 factor1 val1 val3 factor1 val2 val3 factor2 val1 val2 factor2 val1 val3 factor2 val2 val3 ...
and that would be used to calculate correlation coefficients (col1 vs. col2). For the first factor, I would like to get one correlation coefficient for the entire data set (to compare to another published data set for which I would perform the same analysis).
The table for the second factor looks like this:
factor_level col1 col2 factor1 val1 val2 factor1 val1 val3 factor1 val1 val4 factor1 val2 val3 factor1 val3 val4 factor2 val1 val2 factor2 val1 val3 factor2 val2 val3 ...
For the second factor, I would also like to calculate the correlation at each factor level (in the end, I would like to rank the factor levels in my data set by the similarity of their values and design some experiments to test the highest-ranking ones).
After doing this, I got results that seemed to make sense (i.e. the correlations, Spearman's rho, were high in cases where I would expect that based on published data and low where I would expect them to be low).
Out of curiosity, I decided to try another method to calculate correlation coefficients. I randomly split the data at each factor level into 2 groups (if there were 3 rows at a factor level, I left one of them out) and paired up 1 row from each group. The table I used to do the correlation analysis looks like this:
factor_level group1_val group2_val factor1 val1 val1 factor2 val1 val1 factor2 val2 val2 factor2 val3 val3 ...
I repeated this analysis 10 times. That is, I did the random splitting into 2 groups 10 times and calculated the mean Spearman's rho across the 10 runs.
The correlation coefficients using this method were generally lower than what I obtained using the all-pairwise comparisons. I'm sure there is an obvious explanation for this. In a way, I am not disturbed by the differences, because the trends are still the same (i.e. those 'control' cases where I expect high or low correlations are still ranked as expected). And the rankings based on correlation coefficients at the second factor level are similar. My question, though, is whether it is considered appropriate to calculate correlation coefficients as I have done (i.e. using all pairwise comparisons). Or whether it's something other people have done. I don't have 2 variables, so it seems wrong to do it as I have. I'm guessing it might be better to use some form of ANOVA, but I would be happy to use my correlation coefficients so long as their use is not too egregious.
Thank you in advance for your help.