1
$\begingroup$

Why when I try to compute Pearson to a Uniformly distributed observed dataset (the most trivial being: [1, 1, 1]) I get nan?

I'm using python, here is a simple test:

scipy.stats.pearsonr([24 29 47], [1,1,1])

Where [24 29 47] was read from a random number generator.

What I can notice is that Pearson is defined based on $Var(x)$ as in https://stackoverflow.com/questions/7653993/encountered-invalid-value-when-i-use-pearsonr. So how should I reliably compute correlation if I assume that my dataset could be constant?

$\endgroup$
  • 1
    $\begingroup$ Upon running it, I get this error: numpy division with RuntimeWarning: invalid value encountered in double_scalars. So, have a look at this post if it helps. $\endgroup$ – Dawny33 Jun 27 '16 at 6:04
  • 1
    $\begingroup$ You could try adding some very small noise to it so it will compute and be a small correlation. $\endgroup$ – PascalVKooten Jun 27 '16 at 8:49
  • 1
    $\begingroup$ @PascalvKooten Then the result will be completely dependent on the noise you add. I can't see that solution as anything better than arbitrary. $\endgroup$ – Nick Cox Jun 27 '16 at 13:37
  • $\begingroup$ @NickCox For approximation / solving 1 in X bad cases it could be fine. $\endgroup$ – PascalVKooten Jun 27 '16 at 17:46
  • $\begingroup$ The idea of approximation appeals when and only when there is an identifiable solution. What are you approximating here? $\endgroup$ – Nick Cox Jun 27 '16 at 17:50
5
$\begingroup$

Correlation, meaning Pearson correlation, is not defined for such a case. If you consider the definition $\text{cov}(x, y)/[\text{sd}(x) \text{sd}(y)]$ you can see that if either standard deviation is zero, the correlation is indeterminate: as school mathematics underlined, you cannot divide by zero. This isn't a quirk of Python numerics; it's a generic problem.

A more arm-waving way to report the issue is that correlation measures the extent to which there is a linear relationship between variables. But if either variable is a constant, there is no kind of relationship between the variables.

In practice correlations for very small sample sizes are often unreliable any way. In most real situations collecting more data will solve the problem if there is any kind of relationship to quantify.

Uniform distribution is a misnomer in this case, unless a constant distribution is regarded as a degenerate uniform distribution. As your close indicates, the problem is constant data, not uniform data. I've not edited the title or the question, but that emphasis is misleading, indeed wrong.

$\endgroup$
  • $\begingroup$ Indeed, (1,1,1) is not something we would call "uniformly distributed". $\endgroup$ – Richard Hardy Jun 27 '16 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.