# Randomly selected Schools vs. Students - difference in "significant" comparisons

I currently am analyzing student test data for a large school district and we often choose groups of schools to compare test results between - for example the average reading test scores of all charter operated schools vs. traditional district schools. I tend to think that these group selections can be somewhat arbitrary and wanted to compare random groupings of schools to one another to see if we could observe "statistically significant" observations between them.

I wrote an R script to randomly select 25% of schools and then compare the average test scores of students in these schools to the other 75% of schools' students. I also wrote another version that selects 25% of students with no regard to school (25% of student ID numbers) and then compares them to the other 75% of student ID's. I have the script setup up to run the comparison over a large number of iterations.

When I select students by random 25% of schools, I see a large proportion of significant differences between random groupings.

[1] Proportion of Significant Comparisons (alpha = 0.05): 0.647 [1] Proportion of Significant Comparisons (alpha = 0.0033): 0.44

Each group (25% vs 75%) is compared on 15 different subject/grade level tests using a Welch t-test. When I run the scripts again, this time sampling 25% of students regardless of school I see more what I would suspect:

[1] Proportion of Significant Comparisons (alpha = 0.05): 0.053 [1] Proportion of Significant Comparisons (alpha = 0.0033): 0.007

I am not sure what to make of the results of this analysis: What does it say about grouping students by school in this scenario? Seems like random groupings of schools result in a large number of "significant" differences that hold up across multiple iterations of random sampling (>1000).

Would appreciate any thoughts on this - thanks! The design of this was based on the recent ASA's statement on p-value use.

The original analysis was to compare Charter schools vs. District schools and we saw multiple differences between those two groups on their test scores. My thought was along of lines - is there really anything special about Charter schools or could I see similar differences if I just chose schools at random.

• I guess my question is more "Why are there so many significant differences among randomly selected schools than with randomly selected students?" We routinely evaluate groups of schools vs. others (such as Charter vs District), and base our group selection on some underlying criteria (such as school management type). My thought is that drawing conclusions on these groups (Charters have higher test scores than District) isn't reasonable because I could just as easily draw conclusions between comparisons of entirely random groups of schools with no underlying basis for selection.
• When you ran 1, did you compare the students in the schools w/o accounting for schools, or did you just compare the school averages? Mar 21 '16 at 15:46
• Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. Mar 21 '16 at 16:49
• I would do some gross reality checks before making tests. If you make a vioplot of scores by class, what does it look like? By "class" I mean first by school-type, and second by school. link Mar 21 '16 at 17:14
• You should also account for spillover effects here.
– user46925
Mar 21 '16 at 19:25