I have a small sample of marks obtained by a group of students and I would like to apply a binomial test to check if more than half of the size of the classroom has failed due to the application of a new technique for teaching music named X. My data is in vigesimal format and is the following:


The first set of data followed the course by using this new technique, while in the second set are those students that follow the traditional methodology. In this system, a mark of 11 or greater is considered for passing the course.

I have applied hist() and qqnorm() from R and I have the suspicion that the data is not normally distributed.

When I was checking about hypothesis testing I have been reading about the bootstrapping technique for small samples in:


The question that I have is how can I apply bootstrapping by using R to check it up if my hypothesis holds or not. I mean that the new methodology will make more students to fail the subject. Also, I have read about the possibility of doing permutation tests due to the limitations of bootstrapping, but I am really a newbie in the field of statistics.

Any help?

  • $\begingroup$ The bootstrap is a large sample technique, not a small-sample technique $\endgroup$
    – Michael M
    Aug 2, 2017 at 9:19
  • $\begingroup$ @MichaelM there is no reason why permutation test should not be used on relatively small sample. $\endgroup$
    – Tim
    Aug 2, 2017 at 19:28
  • $\begingroup$ Agreed. A permutation test is often the way to go (+1 to your answer). $\endgroup$
    – Michael M
    Aug 2, 2017 at 20:09
  • $\begingroup$ Does my answer help? If something is unclear, feel free to comment. $\endgroup$
    – Tim
    Aug 4, 2017 at 14:47
  • 1
    $\begingroup$ @Layla this kind of analysis only shows you that there is a difference and gives you the probability of such thing happening by chance. You cannot deduce causality from such result, correlation does not imply causation. As about p threshold, the 0.05 etc levels are arbitrary -- your sample is small and it is up to you if you find such value convincing (I'd personally say that it is not that bad given the small sample). $\endgroup$
    – Tim
    Aug 9, 2017 at 8:25

1 Answer 1


Actually, this is a classical scenario to use a permutation test. When using permutation tests, we randomly re-assign the group labels to simulate the "null distribution" (how would your data look like if the group labels were absolutely random). As I understand your question, you have two groups $A$ and $B$, each of the groups took the test, where they passed if their score was at least 11. You want to test the hypothesis that the new method of learning has lead to improvement in test scores, i.e. the proportion of passed exams is greater in group $B$ as compared to $A$.

First, let's change format of your data:

data <- c(data1, data2)
groups <- c(rep("A", length(data1)), rep("B", length(data2)))

Next, let's check the descriptive statistics:

tapply(data, groups, function(y) mean(y >= 11))
##         A         B 
## 0.2800000 0.5263158 

We can see that the proportion of passed tests is greater in group $B$, but is this a significant difference? To check this, let's first define the test statistic:

test <- function(x) diff(tapply(x, groups, function(y) mean(y >= 11)))

We want to look at the difference between proportion of passed exams in group $B$, ($p_B$) and in group $A$ ($p_A$), since $p_B > p_A$ is the same as saying that $p_B - p_A > 0$. So now we can run the simulation:

set.seed(123) # we set the seed for the result to be replicable
res <- replicate(5000, test(sample(data)))

and look at the histogram (the actual result is marked using the red line):

enter image description here

and we check the probabilities

mean(test(data) > res)
## [1] 0.9156

What the test tells us is that in 92% of the cases the actual difference was greater then the differences seen in the "null" data (i.e. data with random labels). So in 92% of cases the results were less extreme then in your case, or saying it differently, if the group labels would be totally random, we could see such results only 8% of the time. So clearly, it made a difference if you were in the "new teaching method" group, or not with $p = 0.08$.

If you want an introductory text that discusses such methods while giving examples (with R code), you can check the following book:

Phillip I. Good . (2013). Introduction to Statistics Through Resampling Methods and R. Wiley.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.