2
$\begingroup$

Here is the question:

Suppose $X, Y$ are independent $N(0,1)$ random variables. And take the regression of $Y$ against $X.$ What is the relationship between $R^2$ and sample size approximately?

First I am not sure if "independent" is a typo in the original question. In ordinary least squares (OLS), there is no specific relation between $R^2$ and sample size.

Here R2 is defined as: $$R2 = 1 - \dfrac{RSS}{TSS}.$$ $$RSS = \sum(y_i-\hat{y})^2,\ TSS = \sum(y_i-\bar{y})^2.$$ And in the simple linear regress with interception, R2 is exactly correlation between X and Y.

$\endgroup$
5
  • 1
    $\begingroup$ Simulate two independent standard normal distributions and calculate the correlation. Is it equal to zero? What happens if you increase the sample size by a factor of ten? $\endgroup$
    – Dave
    Commented Apr 18, 2023 at 2:44
  • $\begingroup$ @Dave I am wrong and updated. Duplicate samples will not change R2. $\endgroup$ Commented Apr 18, 2023 at 5:37
  • $\begingroup$ Simulate cor(rnorm(5,), rnorm(5))^2 in R. Now do it for $10$, then $20$, then $50$, then $100$. $\endgroup$
    – Dave
    Commented Apr 18, 2023 at 10:49
  • $\begingroup$ It seems like you’re using the same definition of $R^2$ that I use in my answer. $\endgroup$
    – Dave
    Commented Apr 18, 2023 at 12:28
  • $\begingroup$ In OLS there is a mathematically predictable relationship between the expected value of $R^2$ and sample size. Moreover, given a sample size you can express the distribution of $R^2$ in the sample in closed form: see the F Ratio distribution $\endgroup$
    – whuber
    Commented Apr 18, 2023 at 12:57

1 Answer 1

3
$\begingroup$

As sample size increases, $R^2$ decreases toward zero.

Since the two variables are independent, they have zero correlation and, thus, zero squared correlation. With a small sample size, some bad luck might result in there being a high empirical correlation. As the sample size increase, however, the empirical correlation will tend toward the true value of zero, hence the squared correlation tending toward zero.

Let’s look at it in a simulation.

library(ggplot2)
set.seed(2023)
Ns <- rep(seq(3, 550, 1), 4) 
r2 <- rep(Ns)
for (i in 1:length(Ns)){
    x <- rnorm(Ns[i])
    y <- rnorm(Ns[i])
    r2[i] <- (cor(x, y))^2
}
d <- data.frame(
    sample_size = Ns,
    r2 = r2
)
ggplot(d, aes(x = sample_size, y = r2)) +
    geom_point()
    xlab(“Sample Size”) +
    ylab(“R^2”)

Squared correlation by sample size

$\endgroup$
12
  • $\begingroup$ why R2 is same as correlation here? I know under OLS assumption, R2 is exactly the correlation between X and Y (with interception). $\endgroup$ Commented Apr 18, 2023 at 12:14
  • $\begingroup$ @user6703592 In simple linear regression (with an intercept), there are a bunch of equivalent definitions of $R^2$, and I picked the one that is easiest to calculate. If you have an equation you prefer, please do post it. Perhaps even try using it in the simulation to see that you get the same trend. // $R^2$ is not the same as correlation here. $R^2$ is the squared correlation, as my code calculates. $\endgroup$
    – Dave
    Commented Apr 18, 2023 at 12:20
  • $\begingroup$ pls see my update. I understand as here R2 is close to the square of correlation but not exactly same. And finally tends to zero? $\endgroup$ Commented Apr 18, 2023 at 12:33
  • $\begingroup$ I can understand it will tend to zero, but I cannot understand under my definition, why it is the empirical correlation between X and Y. Since the cross term is not necessary zero. $\endgroup$ Commented Apr 18, 2023 at 12:35
  • $\begingroup$ Do you mean why the $1-RSS/TSS$ definition equals the squared correlation? $\endgroup$
    – Dave
    Commented Apr 18, 2023 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.