I've run a user-test of a tool where 86 users participated and scored the a according to the System Usability Scale. The scale goes from 0-100, with 0 being the worst and 100 being the best. The gathered results have
The entire dataset can be downloaded here: https://pastebin.com/tdhRJ1by
The scores are not normally distributed, which is expected from the type of test they participated in.
A benchmark score with
σ=21.5exists. Here comes the first question; Can I do a t-test to check whether my sample of test scores are significantly higher than that benchmark score?
T-test with sample means
My intuition tells me that the sample mean which is normally distributed should have the same characteristics of the actual distribution, so I try that. I draw 10,000 samples of 30 without replacement from my sample, giving me a pretty normal looking distribution with
σ=1.98. Using this in an unpaired t-test I get a p value of less than 0.0001, indicating that these are two significantly different populations. However, this seems too convenient - am I massaging the statistics here?
ANOVA with sample means
As a screening question, test users self-identified themselves according to their experience using similar tools. They could choose to answer
a little experienced/ familiar, or
Their distributions are like this, which it's hard to tell anything from:
I try taking means of 10000 samples of size 10 from each of the groups, giving this histogram: I want to know whether these three groups actually have different means, or if they all belong to the same group. So, I calculate the One-Way ANOVA for both the sample means and the original data:
Sample means data:
The sample means data is a way more clear-cut case, does this mean that I've "tampered" with the data?
Can I do these operations using sample means?