Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.

This stackoverflow post describes computing a Pearson correlation of [1,2,3] and [1,5,7] in several different ways in Python. The most straightforward implementation from the definition in Wikipedia comes up with

0.973328526785

while Excel, R, NumPy, an online calculator, and a different Python implementation (involving what looks like a more numerically unstable calculation to me) come up with

0.981980506

I am just curious to know what you think.

share|improve this question
2  
The exact correlation is $\frac{3}{2}\sqrt{\frac{3}{7}} = \sqrt{\frac{27}{28}} \approx$ 0.981 980 506 061 965 716. The value you obtain for the Wikipedia definition looks like a mistake in your implementation: the discrepancy is far too large to be due to numerical instability. (Besides, the Wikipedia formula is the more stable of the two algorithms!) Your result is almost exactly $\sqrt{\frac{18}{19}}$. – whuber Oct 30 '11 at 19:20
6  
I put a comment in the SO post: Beware of the type of the variables! You have encountered an int/float problem. In sum(x) / len(x) you divide ints, not floats. So sum([1,5,7]) / len([1,5,7]) = 13 / 3 = 4, according to integer division (whereas you want 13. / 3. = 4.33...). To fix it rewrite this line as float(sum(x)) / float(len(x)) (one float suffices, as Python converts it automatically). – Piotr Migdal Oct 30 '11 at 19:47
Nice catch, @Piotr! – whuber Oct 30 '11 at 19:48
Nice! If you submit this as an answer, I'll accept it. – dfrankow Oct 30 '11 at 20:00

1 Answer

up vote 10 down vote accepted

(originally written as a comment)

Beware of the type of the variables!

You have encountered an int/float problem. In sum(x) / len(x) you divide ints, not floats. So sum([1,5,7]) / len([1,5,7]) = 13 / 3 = 4, according to the integer division rules (whereas you want 13. / 3. = 4.33...). To fix it rewrite this line as float(sum(x)) / float(len(x)) (one float suffices, as Python converts it automatically).

share|improve this answer
2  
Thanks! Silly bug. Fixed on stackoverflow as well. – dfrankow Oct 30 '11 at 20:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.