Comparing Multi-Year Survey Data (Is Repeated Measures ANOVA appropriate?)

Question

I have a question which requires your helpful expertise.

I am conducting a survey which, through the utilization of a scaled response from participants (1-10), measures for significance in enthusiasm as it pertains to students prior to, and subsequently thereafter completing a course.

This survey was issued on a yearly basis, and as a result of such, with each issuance, the number of individuals differed, as did the individuals themselves.

I would like to compare the results of significance as it pertains to enthusiasm of the students, on a year by year basis.

My first thought was to perform a repeated measures ANOVA, with "YEAR" being the differentiating factor. However, since the sample make-up is dynamic in this case, I'm not sure if they would be appropriate test. However, I remain hesitant to completely rule it out as the samples are more or less drawn from the same student population.

In an internet article from an external website, an individual suggested the utilization of an ANCOVA model, with the co-variate being the initial responses.

I still feel that repeated measures ANOVA is the most appropriate model. Could you please provide me with a guidance as to which model I should utilize and why?

Thanks!

EDIT:

I wanted to provide some further explanation as perhaps my initial explanation caused some nu-necessary confusion.

The data exists in a manner similar to such:

Student ID | Prior Score | Subsequent Score | Year of Course

1 | 10| 10| 2014

2 | 3 | 7 | 2014

3 | 2 | 7 | 2014

3 | 2 | 8 | 2015

4 | 6 | 3 | 2016

5 | 2| 2 | 2016

etc.....

Each individual student was only surveyed once, and the same survey was administered to students each year prior to, and subsequently after completing the course. Therefore, the scores above represent scores pertaining to each survey cycle, differentiated by the rightmost column entry: "Year of Course".

I wanted to measure for a significant increase in score values by year, and then pooled together in a single set, with "Year" being a contributing factor.

Performing a (right tailed) Paired T-Test on each sample, followed by a similar test on the the entire set accomplishes this. However, I wanted to also measure for the impact of the year variable.

This lead me to the repeated measures ANOVA, however, though the population of students does not change by year, the individual identity of the student does. Therefore, repeatedly issuing the survey to different individuals, seemingly violates the assumptions of the test. (Or does it?)

I also was considering performing an ANCOVA, with "Subsequent Score" as the dependent variable, "Prior Score" as the co-variate, and "Year of Course" as the fixed factor.

Any help or guidance that you could provide would be most appreciated. Thanks again for your help.

Answer 1

If you had a series of genuine probability samples taken independently each year then you don't have any sort of repeated measures problem. Each year's data gives you an estimate of that year's student population and you can estimate trends over time by regression (with ANCOVA as one special case).

This sort of independent sampling is approximately what happens with two-year waves of the NHANES series of surveys, and they are indeed analysed as independent samples from their respective populations, using regression for time trends.

If you had non-response with different response biases in different years, that would be a separate problem you'd have to worry about.

Answer 2

If you have some students repeatedly, but most not, then you have ... a big mess, I think. You should have consulted a statistician before starting.

What method to use will depend on how many students are repeated how often. One option is to drop the subsequent responses of any students who are in the survey more than once. Another is to drop those who only appear once. Another is to do some sort of multiple imputation.

Which is best is probably going to be a complex problem and may require more time and effort than can come from this site - you may need to hire an expert.