Is it possible that X and Y are uncorrelated but X can significantly predict Y? [duplicate]

Question

I was just wondering is it possible that X and Y are uncorrelated but X can significantly predict Y?

If so, how would you explain and interpret that?

Answer 1

Yes.

Take for example the unit circle coordinates for $(x, y). $ They are uncorrelated, yet if you know $ x, $ you know that $y$ can only take on $2 $ values - or $1 $ exactly ($y=0$) if $x =1$ or $-1.$

More generally marginal independence does not imply conditional independence. That is there might be a third variable $z$ that allows you to predict $y $ from $x$ knowing $z.$ See Examples of marginal independence, conditional dependence.

Answer 2

Let's prove the fact that @whuber gave in a comment to the question:

Let $X$ be symmetrically distributed around $0$ and let $Y=X^2$. The latter is perfectly predictable from $X$ but the correlation is zero.

We first note that $X$ obviously determines its square $X^2=Y$. The proof of the second part is more interesting:

By definition, a random variable $X$ has a symmetric distribution about $\mu$ if $X-\mu$ has the same distribution as $\mu-X$. For symmetry about $\mu=0$ we get that $X-0=X$ has the same distribution as $0-X=-X$.

Using that $X$ and $-X$ have the same distribution and thus the same moments, the fact that $(-1)^a=-1$ for any odd number $a$, and linearity of expectation, we get $$\mathbb{E}(X^a)=\mathbb{E}[(-X)^a]=\mathbb{E}(-X^a)=-\mathbb{E}(X^a)$$ and hence $\mathbb{E}(X^a)=0$ for any odd number $a$.

For the covariance between $X$ and $Y$ we have $$\mathrm{Cov}(X,Y)=\mathbb{E}(XY)-\mathbb{E}(X)\mathbb{E}(Y)=\mathbb{E}(X^3)-\mathbb{E}(X^1)\mathbb{E}(X^2)$$ and using the previous result $\mathbb{E}(X^a)=0$ for $a\in\{1,3\}$ it follows that $\mathrm{Cov}(X,Y)=0$, which is equivalent to $\mathrm{Corr}(X,Y)=0$.

This is not particularly surprising as the covariance and correlation are measures of linear association between two random variables.

Answer 3

Correlation measures a certain kind of relation (usually linear) between variables, but other relationships (nonlinear) are possible as well. This is discussed in depth in the Why zero correlation does not necessarily imply independence thread. So it is possible that there’s a non-linear relationship between the two variables, that makes them non-independent, yet it is not measured by correlation.

Answer 4

There are a few ways this can happen. Five come to mind. In your case, I suspect the fifth.

1) The relationship between $X$ and $Y$ is nonlinear

As some other answers and comments have discussed, correlation deals with linear relationships. If the relationship is nonlinear, correlation can miss that there is a relationship.

set.seed(2022)
N <- 100
a <- 1
x <- seq(-a, a, 2*a/(N - 1))
y <- x^2
plot(x, y)
cor(x, y)

In this example, $X$ perfectly predicts $Y$. However, the correlation is zero.

The remedy is to consider the quadratic term in a regression, such as L <- lm(y ~ x + I(x^2)). Such a regression gives the expected result of perfect predictability of $Y$.

2) Your WITH and ON functions use different tests

I don't know your software, but there are multiple ways of testing hypotheses. In a theoretical statistics class, such as one taught from the Casella/Berger book Statistical Inference, you will learn about what I call the "big three" hypothesis tests: Wald, likelihood ratio, and score (sometimes called Lagrange multiplier). All of these can give slightly different results. If WITH and ON use different tests of the same null hypothesis, it is possible for one to give a "significant" $p=0.043$ and the other to give an "insignificant" $p = 0.052$.

3) You've included some other variable in the regression

Consider the following picture.

As you might expect, the correlation is zero.

However, I made this by having two groups!

set.seed(2022)
N <- 100
a <- 1
x <- seq(-a, a, 2*a/(N - 1))
g <- rep(c(0, 1), N/2)
y <- x - 2*g*x
plot(x, y)
cor(x, y)

If we consider the group variable g and an interaction between x and g, we get perfect predictability.

L <- lm(y ~ x + g + x*g)

If we know we're dealing with a blue group or a red group, we can make perfect predictions and get perfect correlation.

set.seed(2022)
N <- 100
a <- 1
x <- seq(-a, a, 2*a/(N - 1))
g <- rep(c(0, 1), N/2)
y <- x - 2*g*x
plot(x[g==0], y[g==0], col = 'red')
points(x[g==1], y[g==1], col = 'blue')
plot(x[g==0], y[g==0], col = 'red', main = paste("Red correlation is", cor(x[g==0], y[g==0])))
plot(x[g==1], y[g==1], col = 'blue', main = paste("Blue correlation is", cor(x[g==1], y[g==1])))

4) You have a false positive (type I error)

Hypothesis tests falsely reject true null hypothesis some proportion of the time. Just because you know that, at the population level, $X$ and $Y$ are uncorrelated does not mean that the regression on the sample will reflect that fact.

set.seed(2022)
N <- 250
R <- 1000
ps <- rep(NA, R)
for (i in 1:R){
    
    x <- rnorm(N)
    y <- rnorm(N)
    L <- lm(y ~ x)
    ps[i] <- summary(L)$coef[2, 4] # p-value on the slope coefficient
}

ecdf(ps)(0.05)

Even though we know that $X$ and $Y$ are uncorrelated for each iteration in the above simulation, about $5\%$ of the tests have p-values of $0.05$ or lower, so you have type I errors occur about $5\%$ of the time.

5) You exclude an intercept term, which is what I suspect happened to you

I know one of the common Python regression packages (I forget if it is sklearn or statsmodels) excludes an intercept by default, so Mplus might do the same. Look at the following image of uncorrelated $X$ and $Y$.

If we omit the intercept from the regression, then we force the regression line to go through $(0,0)$.

The regression $\hat y_i = \hat\beta x_i$ has a highly significant $\hat\beta$.

set.seed(2022)
N <- 10000
x <- rnorm(N, 10, 1)
y <- rnorm(N, 10, 1)
plot(x, y)
plot(x, y, xlim = c(0, 14), ylim = c(0, 14))
points(0, 0, col = 'blue')
plot(x, y, xlim = c(0, 14), ylim = c(0, 14))
abline(a = 0, b = 1, col = 'red')
points(0, 0, col = 'blue')
L <- lm(y ~ 0 + x)
summary(L)
cor.test(x, y)

However, when we include the intercept, the slope, correctly, lacks significance.

L <- lm(y ~ x)
summary(L)
Call:
lm(formula = y ~ x)

Residuals:
    Min      1Q  Median      3Q     Max 
-3.8865 -0.6757  0.0123  0.6770  3.7909 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 10.093635   0.099646   101.3   <2e-16 ***
x           -0.007935   0.009924    -0.8    0.424    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9891 on 9998 degrees of freedom
Multiple R-squared:  6.395e-05, Adjusted R-squared:  -3.607e-05 
F-statistic: 0.6394 on 1 and 9998 DF,  p-value: 0.424

HOWEVER, you are correct in that the usual thinking is that correlation and simple linear regression are the same. That is, if $X$ and $Y$ have a correlation that you test somehow (Wald is the usual) and get some p-value $p$, then when you regress $Y$ on $X$ (with an intercept) and test the slope with the same test (Wald is the usual), you should get the same p-value.