Statistical significance for non-random survey

Question

The university I work for is trying to evaluate student satisfaction with undergraduate instruction. For example, we asked if students thought they got enough individual attention and if they were satisfied with the quality of teaching assistants.

We gave the same survey two years in a row but because we tried to survey everyone and not a random sample I’m confused about how to determine if a year to year change is statistically significant or not.

In 2017 we got 19,213 responses to our survey and in 2018 we got 21,185. We attempted to survey the entire student population at the time but not everyone responded to our emails. There were 24,132 eligible students in 2017 and 24,852 eligible students in 2018.

There is some overlap between the respondents in the two years. Of the students that responded either year, about 60% responded both years, 20% responded only in 2017, and 20% only in 2018.

On the survey we had a question where the percent of respondents that responded positively increased by 4 percentage points (from 78% satisfied to 82% satisfied) and another that increased by 2 percentage points (from 85% satisfied to 87% satisfied).

I considered plugging those numbers into a two-sample t test but I think that assumes independent samples, which these are not. I also considered a paired t-test but I think that assumes the same group of subjects both times which isn’t true here. A majority of students overlap but they are not identical and it was an anonymous survey so I can’t figure out exactly who overlaps. Any ideas what method I should use to determine if the 2 or 4 point increases are statically significant or not?

Answer 1

You have a couple different issues going on here: a question about the sampling utilized in your study, and then a question about the appropriate statistical test to use. First to address the sampling question. Random sampling is used when you cannot feasibly study an entire population. It looks like you tried to study the entire population, so I don't think there is a problem with your sampling scheme. However, the issue you are bumping up against is response bias. There might be systematic differences between the satisfaction of the students who responded to your survey and those who did not. Given that you cannot know what the responses would have been of the non-responders, you have to chalk this up as a limitation of your study. But I think that is okay. Most studies on humans come up with this limitation as most settings cannot force participation. Plus it seems like you got a very high response rate, so kudos for that.

Now for the analysis method. You are stuck with assuming these samples are independent because you have no way to link the responses together. If you had a name or a student ID to link student response from one year to the next it would be a different story, but you don't.

However, a t-test is not the correct test for your data. A t-test looks at continuous variables (height, weight, miles, etc.). You have a proportional endpoint, looking at the proportion of students who were satisfied. For that you need to use a chi-square test. Don't worry, it's an easy calculation that you could do by hand if you had to, but there are many online calculators to do it for you.

Finally, your question about whether an increase of 4% is significant. Here's the thing: the larger your sample size, the smaller the "significant" differences will be. The chi-square test will tell you whether there is change in the proportion of satisfied students year over year, but it won't tell you how much of a change matters. Statistics will only answer the question you ask, they won't ensure that you answer the right question. I would argue maybe you are asking the wrong question. Are most students satisfied? Yes. Are the trends in the right direction over time (getting better)? Based on your post my guess is yes. If this survey is done to measure effectiveness of programs or interventions which cost time and effort, how much improvement was anticipated when the program decision was made? Did you achieve that goal? And maybe most importantly, what can you do differently to address the dissatisfied students? Are there qualities of that group that might shed light on why they are dissatisfied (gender, race, educational achievement, income)? These might be more useful questions to get at.