# “Inflated” type I error rate for correlation test when a variable has repeated values

For a given variable Y, if we independently generate a random variable X and test the correlation between Y and X, we know that the histogram of the p-values of the correlation test should show a uniform distribution.

But how about if we have repeated values in X? Say there are 100 unique values of X and they are repeated 20 times. In some cases, we will see a skewed distribution of the p-values.

This can be demonstrated through a simple simulation below. The code is listed at the end.

I am wondering if there is any related reference for this issue of "inflated" type I error rate when testing the correlation while a variable has repeated values. Is there any test developed to fix this inflation, by taking into account the fact that there are repeated values in X? Ideally, for a known repeating mechanism of X, if we randomly generate some unique values of X, the test will produce p-values with a uniform distribution.

Edit: This problem arises from a multivariate regression. We have repeated X for each outcome. We are doing some permutation to test the association between Y and X. When we only permute the unique values of X and keep it repeating across outcomes, we get this issue. We are wondering if we can test on the basis that we know the replication mechanism.

In math language, we hope to find a valid test for $\beta$ in the following simple linear regression

$$Y = X \beta + e = (1_k \otimes \mathbf x) \beta + e$$

when we know $X = 1_k \otimes \mathbf x$, where $X$ is a vector of $n\times k$ elements with $n$ unique values ($\mathbf x$) repeated $k$ times.

This issue can also be interpreted as that if we have repeated values in X, we may tend to observe "spurious correlation" between X and Y.

Edit 2:

Per @Björn's suggestion, I tried introducing random effects, either treating the $n$ samples (each has repeated values of X) or the $k$ outcomes as clusters. It seems the former makes more sense though. It helps, but there is a slight inflation. I calculated the "type I error rates" at the nominal level 0.05, and listed them below in the order of the histograms. Using a normal approximation to calculate their confidence intervals for the 100,000 simulations, we can see the random effects model still produce a type I error rate significantly larger than the nominal level. The code is also updated.

0.04929, 0.07010, 0.05812, 0.05578


set.seed(7)
nsim = 1e5 # it would take a long time to run lmer, may decrease it
pval_cor1 = pval_cor2 = pval_cor3 = pval_cor4 = rep(NA, nsim)
y = rnorm(100*20)
id = factor(rep(1:20, rep(100, 20))) # id for ourcome
id2 = factor(rep(1:100, 20)) # id for sample

for(i in 1:nsim) {
pval_cor1[i] = cor.test(rnorm(100*20), y)$p.value pval_cor2[i] = cor.test(rep(rnorm(100), 20), y)$p.value
tmp = lmer(y ~ x -1  + (x-1|id))
coefs <- data.frame(coef(summary(tmp)))
# use normal distribution to approximate p-value
pval_cor3[i] = 2 * (1 - pnorm(abs(coefs$t.value))) tmp = lmer(y ~ x -1 + (x-1|id2)) coefs <- data.frame(coef(summary(tmp))) pval_cor4[i] = 2 * (1 - pnorm(abs(coefs$t.value)))
}

par(mfrow=c(2,2))
hist(pval_cor1, main = 'Varying X')
hist(pval_cor2, main = 'Repeated X')
hist(pval_cor3, main = 'Repeated X: lmer ID=outcome', xlab = 'p-value')
hist(pval_cor4, main = 'Repeated X: lmer ID=sample', xlab = 'p-value')

• Your simulation is not a valid way to evaluate cor.test. For it to be valid, you need to recreate many different random samples for y, rather than just this one single sample. The reason is that in the second instance, x is not an iid sample. The pattern of its replication is picking out a specific pattern in this particular y. Thus, it should be no surprise that the p-values are tending to be a little small. The test appears to be working as it should. Regardless, x violates the standard distributional assumption of cor.test. – whuber Oct 8 '17 at 21:56
• Thanks a lot @whuber! I have thought about this but did not get this thoroughly. This problem arises from a multivariate regression. We have repeated X for each outcome. We are doing some permutation to test the association between Y and X. When we only permute the unique values of X and keep it repeating across outcomes, we found this issue. We are wondering if we can test on the basis that we know the replication mechanism. BTW, could you also explain a little bit about what do you specifically mean by "a specific pattern in this particular y"? – Randel Oct 8 '17 at 22:33
• Might I suggest including that context and objective in the question itself? That will make the post more readable and perhaps garner really focused and helpful answers. – whuber Oct 9 '17 at 14:01
• This is pretty typical. If you treat correlated values are uncorrelated, everything that follows is invalid. No surprise there. In a regression model you could have a random effect that captures what values come from the same unit/patient/object/etc. – Björn Oct 9 '17 at 14:53
• @Björn, thanks a lot for your suggestion. I tried the random effects. It helps! But there is still a slight inflation. Our model is actually already mixed-effects models and with multiple outcome variables, but we only have outcome-specific random intercepts though. – Randel Oct 9 '17 at 16:23