Unsure which data to use for ANOVA test

Question

I have determined that a one-way ANOVA test of variances will work best for my data. However, the nature of my data makes me very concerned about making any claims based on significance or not.

My research project is examining survey data (25 questions) on a likert-scale taken over a 3 year period, split among 5 subdomains (however, one domain has as little as one corresponding question, another has 8, etc.). I am testing whether or not there is a difference between when respondents took the test at baseline and throughout (7 total time categories, or every 6 months) for EACH SUBDOMAIN.

I am having some trouble saying exactly what the problem is, but simply put I do not know whether I should (a) put every individual response into a bin,count the total responses, and average them at each time slot for each subdomain, or (b) average each individual response, count the number of tests taken during that time slow as an "n", and then take the means and Std.Devs of those for the ANOVA test.

Here are my two tables that I've come up with:

Table for Individual Responses

Table for Grouped Responses

The main difference is that the individual responses has much larger n values. This has direct implications on if my study is significant or not. Here is an example:

and here is why this is important... because it directly matters on whether my data is significant or not:

Results for Individual Responses:

Results for Grouped Responses:

My intuition is telling me I should use option (B) because option A just seems to be artificially increasing the sample size, but again that is the raw data from my responses and I don't want to make any mistakes with this...

If anyone has any good ideas about how to treat this, that would be wonderful news!!! I also think I messed up anyway since I'm using Likert data and I do not know what test is appropriate to use for comparing means of likert data.

Answer 1

In option A you are treating each response that an individual gave as though it came from an independent individual, so yes, your intuition is correct that this is wrong. However, option B may be problematic as well. If I am reading your description correctly, the same individuals responded at multiple time points. If this is current, then that needs to be accounted for as well. However, the changing sample size implies that you have missing data. You would need to use a mixed-effect GLM model to properly address this if my assumption about your data is correct.