I've run a user-test of a tool where 86 users participated and scored the a according to the System Usability Scale. The scale goes from 0-100, with 0 being the worst and 100 being the best. The gathered results have n=86
, μ=74.45
, and σ=13.63
.
The entire dataset can be downloaded here: https://pastebin.com/tdhRJ1by
The distribution of the results when plotted on a frequency histogram looks like this:
The scores are not normally distributed, which is expected from the type of test they participated in.
A benchmark score with n=1000
, μ=68.0
, and σ=21.5
exists. Here comes the first question; Can I do a t-test to check whether my sample of test scores are significantly higher than that benchmark score?
T-test with sample means
My intuition tells me that the sample mean which is normally distributed should have the same characteristics of the actual distribution, so I try that. I draw 10,000 samples of 30 without replacement from my sample, giving me a pretty normal looking distribution with n=10000
, μ=74.43
, and σ=1.98
. Using this in an unpaired t-test I get a p value of less than 0.0001, indicating that these are two significantly different populations. However, this seems too convenient - am I massaging the statistics here?
ANOVA with sample means
As a screening question, test users self-identified themselves according to their experience using similar tools. They could choose to answer totally inexperienced
, a little experienced/ familiar
, or experienced
.
Their distributions are like this, which it's hard to tell anything from:
I try taking means of 10000 samples of size 10 from each of the groups, giving this histogram:
I want to know whether these three groups actually have different means, or if they all belong to the same group. So, I calculate the One-Way ANOVA for both the sample means and the original data:
Original data:
f-measure=4.02
, pvalue=0.021422
.
Sample means data:
f-measure=29034
, pvalue=0.0
.
The sample means data is a way more clear-cut case, does this mean that I've "tampered" with the data?
Can I do these operations using sample means?