Let $X_{ij}$ be data modeled by$X_{ij} \stackrel{\text{iid}}{\sim}N(\theta_j,\sigma_j^2), i=1,2,...,50, j=1,2,...,10$. We test $H_{0j}:\theta_j=0 $ against $H_{1j} : \theta_j\neq0$ for $j=1,2,...,10$ at a level of $\alpha=0.05$. Generate $X_{ij}$ and test the hypothesis, replicate $1000$ times to get the familywise error rate.

I am trying to use R to implement my solution. From my understanding our sample size will be $50$. So we can get the values like so:

r.norm <- rnorm(50)

We want to find the Type-I error (where $H_0$ is true but we reject the null hypothesis). I wrote the below to count how many time we have a false rejection. But that seems incorrect, because our mean will not be equal to $0$, although that is out null hypothesis. Is my logic to calculate FWER correct?

FWER.count <- 0

for(i in 1:1000) {  
    for(j in 1:10) {
        r.norm <- rnorm(50)
        if(mean(r.norm) != 0) {
            if(t.test(r.norm)$p.value <= 0.05) {
                FWER.count <- FWER.count + 1;

FWER <- FWER.count/1000

Edit: Update FWER to have seperate variable for count and the probability. Also added a break to the loop, since we only need to know when one nulls in a group is falsely classified.


1 Answer 1


This my translation of your code snippet.

If the mean of the sample has a mean of zero (it matches the family mean). A t-test is performed. If the p value is less than or equal to $ \alpha $, increment FWER. However, sample mean would only get tested if it matched the mean from the family exactly. The p-values would be 1. The null hypothesis would never get rejected.


FWER would need to be divided by n, to make it a probability. The FWER is the probability of making even one type I error.

To make your code check sample means against the family where the sample mean does not equal zero, then this line.

if(mean(r.norm) == 0)

Should be this.

if(mean(r.norm) != 0)

Why because with a small sample size, I highly doubt the mean would ever be zero. I ran a small test to check.

sapply(rep(50,10), function(x) mean(rnorm(x)))
[1] -0.007272193  0.101159429  0.001891395 -0.138410041 -0.083892761  0.124589734 -0.057765727  0.240548953 -0.072762234 -0.193889501

It gets close, but never zero exactly.

You probably don't even need that first if statement, the mean would rarely be zero and the t-test if statement would ignore any mean that was close to zero.

  • $\begingroup$ I agree that for such a small size it will almost never be equal to zero, but I thought for a type-I error the null had to be true by definition. Yes, I am aware that I need to divide FWER by $n$ to get the probability, I just didn't include it in my code, will update the original post. $\endgroup$
    – Artem
    Dec 13, 2015 at 21:37
  • $\begingroup$ Ah I see. It is my understanding that one accepts null as true (p-value greater than or equal to $ \alpha $) or rejects null as true (p-value less than $ \alpha $). In this case the t-test determines if the null is accepted or rejected based on the significance level chosen. If you look at the wikipedia for FWER it shows declared significant or declared non-signifiant. These would directly relate back to the t-test p-value results. Hope this helps! $\endgroup$
    – Matt L.
    Dec 13, 2015 at 23:53
  • $\begingroup$ Looking at the table on wiki, I am trying to find $V$. The top row "Null hypothesis is True" we check if the group mean is zero or not, and the significance is based on the $\alpha$ level? When I run my code the result seems to make sense, if I add more groups I get more false positives. So I think it works now. $\endgroup$
    – Artem
    Dec 14, 2015 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.