Premise: I have a dataset of elements for which I have a representation in 2 different spaces, a "latent" space and the original space (I can move between those with an Autoencoder neural network).

I want to prove a conjecture where I think that a distance that I developed for the original space doesn't "correlate" with the euclidean distance in the latent space. (By correlate I mean that I feel that "close" elements in one space are "distant" in the other, by their respective distances)

I have computed the distance matrix for both the distances, each in its relative space, for each pair of samples in the dataset.

How do you suggest I can prove this or verify it?


I would use the Pearson correlation coefficient. After doing the test you can look at the p-value to quantify the strength of the correlation.

If you use Python you can use scipy.stats.pearsonr(x, y) to do so, where is x is the distance matrix for one of the spaces and y the other.

Have a look at the documentation here: https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.pearsonr.html

  • $\begingroup$ Thank you, I will try it right away. Anyway, could you provide at least an insight of what this Pearson correlation coefficient is doing? The documentation is not super clear on this. I will let you know how the result looks like. $\endgroup$ Oct 8 '19 at 13:50
  • $\begingroup$ So, first of all the function suggested accepts 1-D inputs, and not the distance matrices. Then even if I do it row wise, I do not know how to interpret the results... For instance this is the output of the first row compared with the first row: (0.009394309417862736, 0.77945516455912) $\endgroup$ Oct 8 '19 at 13:58
  • $\begingroup$ Ok, i didnt recognize that. You can use numpy.ndarray.flatten on both matrices respectively in order to make them 1d. $\endgroup$
    – tamtam_
    Oct 8 '19 at 14:22
  • $\begingroup$ The first output value is the Pearson coefficient [-1, 1]. If it is close to zero, which it is, it indicates no correlation. The p-value also indicates the same. If the p-value is below 0.05, it would indicate a linear correlation, which is not the case here. $\endgroup$
    – tamtam_
    Oct 8 '19 at 14:27
  • $\begingroup$ Ok, thanks. So this is what I get now: pearsonr(D.flatten(), D_Z.flatten()) => (0.011961790971000504, 1.5945123128090787e-26) $\endgroup$ Oct 8 '19 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.