I currently am analyzing student test data for a large school district and we often choose groups of schools to compare test results between - for example the average reading test scores of all charter operated schools vs. traditional district schools. I tend to think that these group selections can be somewhat arbitrary and wanted to compare random groupings of schools to one another to see if we could observe "statistically significant" observations between them.

I wrote an R script to randomly select 25% of schools and then compare the average test scores of students in these schools to the other 75% of schools' students. I also wrote another version that selects 25% of students with no regard to school (25% of student ID numbers) and then compares them to the other 75% of student ID's. I have the script setup up to run the comparison over a large number of iterations.

When I select students by random 25% of schools, I see a large proportion of significant differences between random groupings.

[1] Proportion of Significant Comparisons (alpha = 0.05): 0.647 [1] Proportion of Significant Comparisons (alpha = 0.0033): 0.44

Each group (25% vs 75%) is compared on 15 different subject/grade level tests using a Welch t-test. When I run the scripts again, this time sampling 25% of students regardless of school I see more what I would suspect:

[1] Proportion of Significant Comparisons (alpha = 0.05): 0.053 [1] Proportion of Significant Comparisons (alpha = 0.0033): 0.007

I am not sure what to make of the results of this analysis: What does it say about grouping students by school in this scenario? Seems like random groupings of schools result in a large number of "significant" differences that hold up across multiple iterations of random sampling (>1000).

Would appreciate any thoughts on this - thanks! The design of this was based on the recent ASA's statement on p-value use.

The original analysis was to compare Charter schools vs. District schools and we saw multiple differences between those two groups on their test scores. My thought was along of lines - is there really anything special about Charter schools or could I see similar differences if I just chose schools at random.

  • I guess my question is more "Why are there so many significant differences among randomly selected schools than with randomly selected students?" We routinely evaluate groups of schools vs. others (such as Charter vs District), and base our group selection on some underlying criteria (such as school management type). My thought is that drawing conclusions on these groups (Charters have higher test scores than District) isn't reasonable because I could just as easily draw conclusions between comparisons of entirely random groups of schools with no underlying basis for selection.
  • $\begingroup$ When you ran 1, did you compare the students in the schools w/o accounting for schools, or did you just compare the school averages? $\endgroup$ Mar 21, 2016 at 15:46
  • 2
    $\begingroup$ Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. $\endgroup$ Mar 21, 2016 at 16:49
  • $\begingroup$ I would do some gross reality checks before making tests. If you make a vioplot of scores by class, what does it look like? By "class" I mean first by school-type, and second by school. link $\endgroup$ Mar 21, 2016 at 17:14
  • $\begingroup$ You should also account for spillover effects here. $\endgroup$
    – user46925
    Mar 21, 2016 at 19:25

1 Answer 1


When you sample random schools, you are comparing school-level averages, not just student-level averages.* The robustness of the effect in this case indicates that the school-level averages in your dataset are probably not normally distributed.

On the other hand, when you sample the students independently of school, you are no longer looking at individual school averages. Since you see no effect here, this indicates that your individual student scores probably are normally distributed.

So, you are seeing differences here because you are fundamentally testing two different hypotheses.

*Edit: Here's an example illustrating why this is the case: Take two schools A and B. A has scores (2, 4, 5, 6, 7, 8), B has scores (3, 4, 2, 1). The mean of A is 5.33, of B is 2.5, and of the whole dataset is 4.2. There are 6 entries in A and 4 in B, so the weighted average of A and B would be (6/10)*mean(A) + (4/10)*mean(B) = 0.6*5.33 + 0.4*2.5 = 4.2, the same as the whole dataset mean.

  • $\begingroup$ School averages are not compared. Students are grouped either randomly by ID or randomly by what school # they attend. Results are then averaged and compared for each group (75% vs. 25%). So it's either a group of students from random schools or a random group of students, but the group averages are calculated from the student-level data. $\endgroup$
    – crw636
    Mar 21, 2016 at 15:54
  • $\begingroup$ Even though you are calculating the averages from student level data, you are still essentially performing a weighted averaging of the school level data. Take two schools A and B. A has scores (2, 4, 5, 6, 7, 8), B has scores (3, 4, 2, 1). The mean of A is 5.33, of B is 2.5, and of the whole dataset is 4.2. There are 6 entries in A and 4 in B, so the weighted average of A and B would be (6/10)*mean(A) + (4/10)*mean(B) = 0.6*5.33 + 0.4*2.5 = 4.2. This sampling method implicitly includes school level information. $\endgroup$ Mar 21, 2016 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.