I have three samples of size $N=10^6$ for the three responses $Y_1,Y_2,Y_3$ (or one sample of size $N$ for the random vector $\mathbf {Y}$). For each response the sample is i.i.d., even if the three variables are correlated. The distributions look quite clearly non-normal to me (just look how skewed they are): enter image description here enter image description here enter image description here

However, this can be subjective (someone else said that they look Gaussian). I could have computed sample skewness and kurtosis, but I don't have formulas for the standard errors, even though I could have used bootstrap, given that $N$ is so large. Instead, I chose to simply compute p-values using the Anderson-Darling composite test, and of course I got p < 2.2e-16 for each response. Note that the normality test is not done for the sake of applying other statistical methods, in this case. I just wanted to show that these variables were not normally distributed.

Now, in this specific case I don't need to make any adjustment for multiplicity, because the p-values are so low that with any correction, the tests would still reject normality. But suppose I had much larger p-values and I wanted to adjust for complexity. The three variables are quite correlated: $$\operatorname{cor}(Y_1,Y_2)= 0.330, \quad \operatorname{cor}(Y_1,Y_3)= 0.935, \quad \operatorname{cor}(Y_2,Y_3)= 0.642$$

Qualitatively, I expect that since $Y_1, Y_2, Y_3$ are positively correlated, the probability of having a statistically significant result when the three nulls are true, is closer to that of a single variable, than when $Y_1, Y_2, Y_3$ are independent. As a matter of fact, in the limiting case of correlation $= 1$, the probability is the same and no multiplicity correction is required.

Quantitatively, which would be the correct adjustment for multiplicity in this case? Can you give me a possibly simple mathematical explanation of your answer? I'd like to use one of the methods of the p.adjust function in R, if possible. Otherwise, other methods are also accepted, but I would still like to understand the answer from the point of view of probability theory/mathematical statistics, if it's not something very complicated.

  • $\begingroup$ You don't have $10^6$ samples; you have one sample consisting of $10^6$ observations or data points. $\qquad$ $\endgroup$ – Michael Hardy Jul 3 '16 at 19:38
  • $\begingroup$ @MichaelHardy, I'm nearly convinced: a sample of size $N$ sounds right. However, how would I refer to the i.i.d. concept? Data points are not i.i.d. (they are just realizations of random variables, not random variables). So, should I talk of an i.i.d. sample? Meaning, a realization from a vector of $N$ i.i.d. random variables? $\endgroup$ – DeltaIV Jul 3 '16 at 20:08
  • $\begingroup$ "realization of an i.i.d. sample" is correct. $\qquad$ $\endgroup$ – Michael Hardy Jul 3 '16 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.