# Statistical significance for non-random survey

The university I work for is trying to evaluate student satisfaction with undergraduate instruction. For example, we asked if students thought they got enough individual attention and if they were satisfied with the quality of teaching assistants.

We gave the same survey two years in a row but because we tried to survey everyone and not a random sample I’m confused about how to determine if a year to year change is statistically significant or not.

In 2017 we got 19,213 responses to our survey and in 2018 we got 21,185. We attempted to survey the entire student population at the time but not everyone responded to our emails. There were 24,132 eligible students in 2017 and 24,852 eligible students in 2018.

There is some overlap between the respondents in the two years. Of the students that responded either year, about 60% responded both years, 20% responded only in 2017, and 20% only in 2018.

On the survey we had a question where the percent of respondents that responded positively increased by 4 percentage points (from 78% satisfied to 82% satisfied) and another that increased by 2 percentage points (from 85% satisfied to 87% satisfied).

I considered plugging those numbers into a two-sample t test but I think that assumes independent samples, which these are not. I also considered a paired t-test but I think that assumes the same group of subjects both times which isn’t true here. A majority of students overlap but they are not identical and it was an anonymous survey so I can’t figure out exactly who overlaps. Any ideas what method I should use to determine if the 2 or 4 point increases are statically significant or not?

• Randomization prevents bias. Without randomization you can't be sure about bias and so you can't do statistical inference. – Michael R. Chernick May 26 '18 at 1:10

You have a couple different issues going on here: a question about the sampling utilized in your study, and then a question about the appropriate statistical test to use. First to address the sampling question. Random sampling is used when you cannot feasibly study an entire population. It looks like you tried to study the entire population, so I don't think there is a problem with your sampling scheme. However, the issue you are bumping up against is response bias. There might be systematic differences between the satisfaction of the students who responded to your survey and those who did not. Given that you cannot know what the responses would have been of the non-responders, you have to chalk this up as a limitation of your study. But I think that is okay. Most studies on humans come up with this limitation as most settings cannot force participation. Plus it seems like you got a very high response rate, so kudos for that.

Now for the analysis method. You are stuck with assuming these samples are independent because you have no way to link the responses together. If you had a name or a student ID to link student response from one year to the next it would be a different story, but you don't.

However, a t-test is not the correct test for your data. A t-test looks at continuous variables (height, weight, miles, etc.). You have a proportional endpoint, looking at the proportion of students who were satisfied. For that you need to use a chi-square test. Don't worry, it's an easy calculation that you could do by hand if you had to, but there are many online calculators to do it for you.

Finally, your question about whether an increase of 4% is significant. Here's the thing: the larger your sample size, the smaller the "significant" differences will be. The chi-square test will tell you whether there is change in the proportion of satisfied students year over year, but it won't tell you how much of a change matters. Statistics will only answer the question you ask, they won't ensure that you answer the right question. I would argue maybe you are asking the wrong question. Are most students satisfied? Yes. Are the trends in the right direction over time (getting better)? Based on your post my guess is yes. If this survey is done to measure effectiveness of programs or interventions which cost time and effort, how much improvement was anticipated when the program decision was made? Did you achieve that goal? And maybe most importantly, what can you do differently to address the dissatisfied students? Are there qualities of that group that might shed light on why they are dissatisfied (gender, race, educational achievement, income)? These might be more useful questions to get at.

• Thanks very much - I really appreciate the comments on methods and conclusions. – BraceBirdWise May 26 '18 at 17:22
• Am I doing this right for the chi-squared test? I have n of 19213 in 2017 and 21185 in 2018. And I'm testing a satisfaction increase from 85% to 87%. My null hypothesis is that the true level of satisfaction is the same in 2017 and 2018. So I set up the calculation like this imgur.com/a/SQGkp8i That gives me a small p-value of 7e-9. – BraceBirdWise May 26 '18 at 17:43
• So that means that if it were really true that the group that answered the survey was equally satisfied both years, it would be very unlikely to see this level of difference in the results. However, because my response rate is only 85% and those missing 15% may be very different since they are non-random that means that the respondents may not be representative of the entire population. – BraceBirdWise May 26 '18 at 17:48
• And my chi-squared test assumes the two samples are independent which in this case they aren't - about 60% of the students overlap.So that means (what?) that the probably estimate may be lower than it would be if the samples were truly independent? – BraceBirdWise May 26 '18 at 17:48
• It looks like you are doing the test correctly. As I said, with a sample size that large, it is pretty easy to achieve statistical significance. You are correct that the missing respondents may be systematically different from the respondents, and if you somehow knew their answers you might get a different result. However, there is no way to statistically address this given the data you have. – Bosley May 29 '18 at 13:04