# Trying to determine the similarity of the distributions of labels

I asked this question on stackoverflow first since it is more of an implementation question, but I got no answers yet, therefore I'm asking it here again.

I have a dataset which is annotated by 2 different sources. This means that for every sample, 2 different sources give a label. I want to see how similar the two sets of labels are. Therefore I thought of using the Spearman correlation and the test for homogeneity with chi squared.

I made this question because it is the first time I'm doing this and also in Python. So I'm not sure I'm doing them right, and would like some feedback.

For example I have my 2 sets:

set1 = [1,5,7,3,2,4,...]
set1 = [1,3,7,2,2,1,...]
rho, pval = scipy.stats.spearmanr(set1,set2)


The results are:

rho = 0.456498145813
p value = 0.0


What do these results mean? (I think there is very little correlation, though why do I have a $p$ value of 0?)

And for the test for homogeneity I calculate the frequencies and use the chi squared test:

chisq , p = scipy.stats.chisquare(np.array(distributionSet1),np.array(distributionSet2),6)


The results here are:

chisq = 47611.7764023
p = nan


How can the $p$ value be nan? And what does such a high chi squared value mean?

For the degrees of freedom, I calculated 6 (I have 2 sets, so 2 populations, and 7 categories, so $(2-1)\times(7-1)=6$).

Not sure why Python would tell you the $p$ value for that $\chi^2$ value is not a number (nan), but the high $\chi^2$ value probably means that whatever model you fit to your label frequencies didn't fit well at all. The usual $\chi^2$ test of homogeneity fits a model of identical expected values (frequencies in your case) to all categories, so if you didn't mess with the defaults at all, it's probably telling you that your labels appear with different frequencies (some more often than others) in your dataset. Edit: This result applies to both of your sources; it is an omnibus test, meaning it tests the fit of your model to all your different categories (labels) in both sources. It's impossible to tell just from your $\chi^2$ value whether the fit is poor because one of your sources uses its labels unevenly (regardless of whether the other does), or whether it's because your two sources use their labels differently from one another (regardless of whether each source uses its labels evenly).
The $\rho$ value you calculated is not what we'd generally call "very little." It's roughly equivalent to a Pearson's $r = .47$ (see my related answer incorporating Gilpin's (1993) method of converting $\rho$ to $r$), which is fairly strong. If your sample sizes are large enough, you can expect your estimate of a strong correlation to differ significantly from zero, which is what your $p$ value for $\rho = .46$ is telling you: it's significantly different from zero. Edit: If you did everything right, this should mean that your two sources used the same labels as each other pretty often (regardless of whether each source used its labels with equal frequency).