The university I work for is trying to evaluate student satisfaction with undergraduate instruction. For example, we asked if students thought they got enough individual attention and if they were satisfied with the quality of teaching assistants.
We gave the same survey two years in a row but because we tried to survey everyone and not a random sample I’m confused about how to determine if a year to year change is statistically significant or not.
In 2017 we got 19,213 responses to our survey and in 2018 we got 21,185. We attempted to survey the entire student population at the time but not everyone responded to our emails. There were 24,132 eligible students in 2017 and 24,852 eligible students in 2018.
There is some overlap between the respondents in the two years. Of the students that responded either year, about 60% responded both years, 20% responded only in 2017, and 20% only in 2018.
On the survey we had a question where the percent of respondents that responded positively increased by 4 percentage points (from 78% satisfied to 82% satisfied) and another that increased by 2 percentage points (from 85% satisfied to 87% satisfied).
I considered plugging those numbers into a two-sample t test but I think that assumes independent samples, which these are not. I also considered a paired t-test but I think that assumes the same group of subjects both times which isn’t true here. A majority of students overlap but they are not identical and it was an anonymous survey so I can’t figure out exactly who overlaps. Any ideas what method I should use to determine if the 2 or 4 point increases are statically significant or not?