1
$\begingroup$

It's well known that Pearson correlation can underestimate the true value in case of non-linear relationship. But can it overestimate? For example, Pearson correlation of discontinuous distributions, shown here (no. 5), heavily depends on the outliers. This is the only example I found where linear (Pearson) correlation overestimates the intuitive relation.

But if there are no outliers, can Pearson correlation overestimate the true value? The true value is given by an oracle that outputs (nonlinear) correlation for any two complex distributions (or their infinite realizations X and Y). One of these oracles can be Shannon mutual information.

Consider two random variables $X$ and $Y$, drawn from Bernoulli distribution with probability $p$ and coupling such that $P(y=1|x=1)=1-p$:

from sklearn.metrics import normalized_mutual_info_score

def coupled_bern(p=0.1, n=100000): 
    x = np.random.binomial(n=1, p=p, size=n) 
    py = np.where(x, 1-p, p) 
    y = np.random.binomial(n=1, p=py, size=n) 
    return x, y

x, y = coupled_bern()
nonlinear_corr = normalized_mutual_info_score(x, y, average_method='arithmetic')  # 0.37
corr = np.corrcoef(x,y)[0,1]  # 0.62

In this example, for any $p$, the linear correlation is always higher than nonlinear (normalized MI). However, I'm not sure that we can say that "linear correlation overestimates the correct value" because we don't know the "correct" value, which is something undefined here.


The question I posed in the title comes from the paper, where the authors claim "when the relation between the two signals is nonlinear, magnitude-squared coherence (MSC) can produce totally misleading results". MSC can be viewed as a linear correlation estimator in the frequency domain. And the plots they have for MSC demonstrate overestimated coherence (read correlation), compared to mutual information.

$\endgroup$
  • 1
    $\begingroup$ The crunch here is that, as you hint, what is true or correct must be determined independently to allow false or incorrect correlations to be identified. However, it is entirely appropriate to stress to learners that correlations can be surprisingly or misleadingly high or low, e.g. when outliers pull correlations up or down. The correlation, there and elsewhere, is not false or incorrect: it is just doing what its definition implies. $\endgroup$ – Nick Cox Dec 19 '19 at 11:44
  • $\begingroup$ Further, many of who think they aren't learners can be caught out. I've known experienced researchers in some fields think themselves above looking at scatter plots and so miss a signal about their data that is obscured by the correlations. Many would be surprised at how high correlations can be between different powers of a variable even though they know that the relationship isn't linear. $\endgroup$ – Nick Cox Dec 19 '19 at 11:45
  • $\begingroup$ I like a lot the figure on the top right of the wikipedia page on correlation and the figure on the section "correlation and linearity". The 2nd comment made me think about Hermite polynomials. $\endgroup$ – Ertxiem - reinstate Monica Dec 19 '19 at 11:50
  • $\begingroup$ This question has an air of tautology about it: what, exactly, would an "outlier" be in this situation? It can only be a situation that could cause some particular correlation estimator to have a positive bias. The other issue with this question is that it doesn't even have a target of estimation--the correct value is undefined--which makes it impossible to determine what is being asked. $\endgroup$ – whuber Dec 19 '19 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.