Why is the pearson correlation between the 40 and 60 percentile much lower than with the full range?

Question

Intuitively given the definition of pearson correlation coefficient, I thought at first that partitioning the full range to the data between the 20 and 40 percentile (closer to the median/mean for gaussian distributions) would yield a higher pearson correlation coefficient. Actually, is the other way round.

Why is this?

See the following example:

import numpy as np
from scipy.stats import pearsonr    

mu_x=3
sigma_x=0.01
x = np.random.normal(mu_x, sigma_x, 100000)

mu_y= 0 
sigma_y = 0.1
y = 3*x + 0.2*np.random.normal(mu_y, sigma_y, 100000)

pearsonr(x,y)
(0.83099291398880859, 0.0)

x2=x[ np.where( (x >= np.percentile(x,40)) & (x <= np.percentile(x,60)))]
y2=y[ np.where( (y >= np.percentile(y,40)) & (y <= np.percentile(y,60)))]

pearsonr(x2,y2)

(-0.0029887929213811139, 0.67254795967257908)

Added this following @NickCox suggestion:

for p in range(0,50):
 x2=x[ np.where( (x >= np.percentile(x,p)) & (x <= np.percentile(x,100-p)))]
 y2=y[ np.where( (y >= np.percentile(y,p)) & (y <= np.percentile(y,100-p)))]
 print p,100-p,pearsonr(x2,y2)

 0 100 (0.83235076737608205, 0.0)
 1 99 (0.022661699597148071, 1.2930035229909938e-12)
 2 98 (0.0080658659891794712, 0.01245002894257673)
 .
 .
 .
 continue oscillating around zero.

Answer 1

If you truly have a linear association, Pearson's correlation will get smaller when you subset to smaller ranges. This is a classic way to attenuate the estimated slope of a line and a pitfall to watch out for when thinking about subgroup analyses.

# Pearson's correlation attentuated with subsampling
# example in R
set.seed(1)
library(MASS)
Sigma <- matrix(c(10,3,3,2),2,2)
Sigma
d <- mvrnorm(n = 1000, rep(0, 2), Sigma)
colnames(d)<-c('x','y')
dim(d)
head(d)
plot(d)
cor.test(d[,1],d[,2]) # r = 0.67

# just trimming on X starkly reduces the observed correlation
xTrimmed <- (d[,1] > quantile(d[,1],0.25) & d[,1] < quantile(d[,1],0.75))
dTrimX <- d[xTrimmed,] 
dim(dTrimX)
plot(dTrimX)
cor.test(dTrimX[,1],dTrimX[,2]) # r = 0.29

# further trimming on Y does more so
xyTrimmed <- (dTrimX[,2] > quantile(dTrimX[,2],0.25) & dTrimX[,2] < quantile(d[,1],0.75))
dTrimXY <- dTrimX[xyTrimmed,] 
dim(dTrimXY)
plot(dTrimXY)
cor.test(dTrimXY[,1],dTrimXY[,2]) # r = 0.21

# quick plot to illustrate what's happening
plot(d, xlim=c(min(d[,1]),max(d[,1])), , ylim=c(min(d[,2]),max(d[,2])), col='blue')
abline( lm(d[,2]~d[,1]), col='blue' )
par(new=T)
plot(dTrimXY, xlim=c(min(d[,1]),max(d[,1])), , ylim=c(min(d[,2]),max(d[,2])), col='red')
abline( lm(dTrimXY[,2]~dTrimXY[,1]), col='red' )

However, here is a case where the trimming reveals a lack of association for all other points.

# generate data with no correlation
set.seed(8)
x <- rnorm(20)
y <- rnorm(20)
cor.test(x,y) # r = 0.003
# add two outliers
x[21] <- y[21] <- -7
x[22] <- y[22] <- 7
plot(x,y)
abline(lm(y~x))
cor.test(x,y) # r = 0.82

Answer 2

I think that there is a bug in the example code of the question.

You can not compare x2 and y2 after

x2=x[ np.where( (x >= np.percentile(x,40)) & (x <= np.percentile(x,60)))]
y2=y[ np.where( (y >= np.percentile(y,40)) & (y <= np.percentile(y,60)))]

since they are no more "sincronized" as x and y were.

np.where return a subset of indices, the two subset for x and y are not guaranteed to be equal (usually they are not equal), then you probably lose the correlation because you do not compare anymore the same array components as before.