Randomly selected Schools vs. Students - difference in "significant" comparisons

Question

I currently am analyzing student test data for a large school district and we often choose groups of schools to compare test results between - for example the average reading test scores of all charter operated schools vs. traditional district schools. I tend to think that these group selections can be somewhat arbitrary and wanted to compare random groupings of schools to one another to see if we could observe "statistically significant" observations between them.

I wrote an R script to randomly select 25% of schools and then compare the average test scores of students in these schools to the other 75% of schools' students. I also wrote another version that selects 25% of students with no regard to school (25% of student ID numbers) and then compares them to the other 75% of student ID's. I have the script setup up to run the comparison over a large number of iterations.

When I select students by random 25% of schools, I see a large proportion of significant differences between random groupings.

[1] Proportion of Significant Comparisons (alpha = 0.05): 0.647 [1] Proportion of Significant Comparisons (alpha = 0.0033): 0.44

Each group (25% vs 75%) is compared on 15 different subject/grade level tests using a Welch t-test. When I run the scripts again, this time sampling 25% of students regardless of school I see more what I would suspect:

[1] Proportion of Significant Comparisons (alpha = 0.05): 0.053 [1] Proportion of Significant Comparisons (alpha = 0.0033): 0.007

I am not sure what to make of the results of this analysis: What does it say about grouping students by school in this scenario? Seems like random groupings of schools result in a large number of "significant" differences that hold up across multiple iterations of random sampling (>1000).

Would appreciate any thoughts on this - thanks! The design of this was based on the recent ASA's statement on p-value use.

The original analysis was to compare Charter schools vs. District schools and we saw multiple differences between those two groups on their test scores. My thought was along of lines - is there really anything special about Charter schools or could I see similar differences if I just chose schools at random.

• I guess my question is more "Why are there so many significant differences among randomly selected schools than with randomly selected students?" We routinely evaluate groups of schools vs. others (such as Charter vs District), and base our group selection on some underlying criteria (such as school management type). My thought is that drawing conclusions on these groups (Charters have higher test scores than District) isn't reasonable because I could just as easily draw conclusions between comparisons of entirely random groups of schools with no underlying basis for selection.

Answer 1

When you sample random schools, you are comparing school-level averages, not just student-level averages.* The robustness of the effect in this case indicates that the school-level averages in your dataset are probably not normally distributed.

On the other hand, when you sample the students independently of school, you are no longer looking at individual school averages. Since you see no effect here, this indicates that your individual student scores probably are normally distributed.

So, you are seeing differences here because you are fundamentally testing two different hypotheses.

*Edit: Here's an example illustrating why this is the case: Take two schools A and B. A has scores (2, 4, 5, 6, 7, 8), B has scores (3, 4, 2, 1). The mean of A is 5.33, of B is 2.5, and of the whole dataset is 4.2. There are 6 entries in A and 4 in B, so the weighted average of A and B would be (6/10)*mean(A) + (4/10)*mean(B) = 0.6*5.33 + 0.4*2.5 = 4.2, the same as the whole dataset mean.