0
$\begingroup$

I have roughly 30 surveys (one for each of 30 schools). They all asked the same Likert item questions. For each item, I want to know if any of the results are significant. That is, I want to know if any of the schools stand out among the others either positively or negatively. What should I do?

We asked parents from 30 different schools 20 questions about their school. I have made bar charts of the schools on the same graph so I can see generally how the schools compare to each other. I could rank order the schools based on their top box or top two box scores for each question. I guess I thought I could figure out if any of the schools did significantly better or worse than any others using some test of statistical significance.

Most schools look pretty similar on any given item and I would suspect that their scores are "the same". I want to know how different scores would have to be to be considered different

$\endgroup$
6
  • $\begingroup$ This is rather broad & ambiguous. Are you saying you want to test every item on your questionnaires individually? What are the study units (eg, patients or participants)? Can you post your data? This really doesn't make much sense right now. $\endgroup$ May 8 '17 at 17:44
  • $\begingroup$ We asked parents from 30 different schools 20 questions about their school. I have made bar charts of the schools on the same graph so I can see generally how the schools compare to each other. I could rank order the schools based on their top box or top two box scores for each question. I guess I thought I could figure out if any of the schools did significantly better or worse than any others using some test of statistical significance. $\endgroup$
    – bbf
    May 8 '17 at 17:49
  • $\begingroup$ Most schools look pretty similar on any given item and I would suspect that their scores are "the same". I want to know how different scores would have to be to be considered different. $\endgroup$
    – bbf
    May 8 '17 at 17:52
  • $\begingroup$ OK, that helps (a lot, actually). How many parents / respondents do you have per school? (Also, please edit the clarifying information into the body of your question.) $\endgroup$ May 8 '17 at 18:28
  • $\begingroup$ varies from school to school one is very low like 17 but most are around 200. $\endgroup$
    – bbf
    May 8 '17 at 19:07
1
$\begingroup$

If you are interested in just one question, and are comparing just two schools, and you want to know whether the observed difference is statistically significant, the standard answer would be to do a Mann-Whitney test. (Mann-Whitney is a non-parametric version of the t-test.)

If you are interested in just one question, and are comparing more than two schools, and you want to know whether at least one school is rated differently from the others in a statistically significant way, the standard answer would be to do a Kruskal-Wallis test. (Kruskal-Wallis is a non-parametric version of single-factor ANOVA, which is a generalization of the t-test to more than two groups.)

If you are interested in doing this for all the questions, you are performing multiple comparisons, so you will need to make an adjustment to your critical probability, such as a Bonferroni correction, to keep the chance of a false positive low. (If you do a test on 20 questions using a critical probability of 0.05 on each, you have a 65% chance of getting a false positive.)

That said, with all this data and no particular question in mind, it sound like you are going on a fishing expedition. It's very difficult to do statistics right on a fishing expedition, and you are very likely to end up finding things that end up being not replicated when the specific assertions are tested in independent follow-ups. If you really want to go on a fishing expedition, the best way to do so is probably to segregate the data at the outset into two groups. Use the first group of the data for exploratory analysis. Based on what you find from that data, formulate a few specific questions and test them on the remaining data, which should have been kept entirely hidden to you during the exploratory phase.

$\endgroup$
1
  • $\begingroup$ I think I am interested in the second (one question, and are comparing more than two schools). I am fishing in that I want to fish for any problems at the schools. I thought this would be more fair than just saying the schools that scored in the bottom 3 are "bad" for example (if in fact there is no significant differences in the schools). I guess the problem is with not knowing what score is good enough so I was deciding that bad is "different from everyone else" does that make sense? I did not design the survey,I am just tasked to provide feedback from it. $\endgroup$
    – bbf
    May 8 '17 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.