Trying to determine the similarity of the distributions of labels

Question

I asked this question on stackoverflow first since it is more of an implementation question, but I got no answers yet, therefore I'm asking it here again.

I have a dataset which is annotated by 2 different sources. This means that for every sample, 2 different sources give a label. I want to see how similar the two sets of labels are. Therefore I thought of using the Spearman correlation and the test for homogeneity with chi squared.

I made this question because it is the first time I'm doing this and also in Python. So I'm not sure I'm doing them right, and would like some feedback.

For example I have my 2 sets:

set1 = [1,5,7,3,2,4,...]
set1 = [1,3,7,2,2,1,...]
rho, pval = scipy.stats.spearmanr(set1,set2)

The results are:

rho = 0.456498145813
p value = 0.0

What do these results mean? (I think there is very little correlation, though why do I have a $p$ value of 0?)

And for the test for homogeneity I calculate the frequencies and use the chi squared test:

chisq , p = scipy.stats.chisquare(np.array(distributionSet1),np.array(distributionSet2),6)

The results here are:

chisq = 47611.7764023
p = nan

How can the $p$ value be nan? And what does such a high chi squared value mean?

For the degrees of freedom, I calculated 6 (I have 2 sets, so 2 populations, and 7 categories, so $(2-1)\times(7-1)=6$).

Answer 1

Not sure why Python would tell you the $p$ value for that $\chi^2$ value is not a number (nan), but the high $\chi^2$ value probably means that whatever model you fit to your label frequencies didn't fit well at all. The usual $\chi^2$ test of homogeneity fits a model of identical expected values (frequencies in your case) to all categories, so if you didn't mess with the defaults at all, it's probably telling you that your labels appear with different frequencies (some more often than others) in your dataset. Edit: This result applies to both of your sources; it is an omnibus test, meaning it tests the fit of your model to all your different categories (labels) in both sources. It's impossible to tell just from your $\chi^2$ value whether the fit is poor because one of your sources uses its labels unevenly (regardless of whether the other does), or whether it's because your two sources use their labels differently from one another (regardless of whether each source uses its labels evenly).

The $\rho$ value you calculated is not what we'd generally call "very little." It's roughly equivalent to a Pearson's $r = .47$ (see my related answer incorporating Gilpin's (1993) method of converting $\rho$ to $r$), which is fairly strong. If your sample sizes are large enough, you can expect your estimate of a strong correlation to differ significantly from zero, which is what your $p$ value for $\rho = .46$ is telling you: it's significantly different from zero. Edit: If you did everything right, this should mean that your two sources used the same labels as each other pretty often (regardless of whether each source used its labels with equal frequency).

If your labels are nominal, I don't really see why you couldn't just calculate the percentage of label pairs (from both of your sources, each labeling the same observation/criterion) that agree. That's how I've handled judge agreement in my own nominal coding tasks. I'd love to see any alternative suggestions from other answerers though! For instance, it might be nice to see an alternative that could take into account "near misses" if some non-identical label pairs are still more similar than others, but not in such a way as to suit a ranking scheme by which all labels could be ordered from first to last.