Can I use the sample mean to check whether non-normally distributed populations are different?

Question

I've run a user-test of a tool where 86 users participated and scored the a according to the System Usability Scale. The scale goes from 0-100, with 0 being the worst and 100 being the best. The gathered results have n=86, μ=74.45, and σ=13.63.

The entire dataset can be downloaded here: https://pastebin.com/tdhRJ1by

The distribution of the results when plotted on a frequency histogram looks like this:

The scores are not normally distributed, which is expected from the type of test they participated in.

A benchmark score with n=1000, μ=68.0, and σ=21.5exists. Here comes the first question; Can I do a t-test to check whether my sample of test scores are significantly higher than that benchmark score?

T-test with sample means

My intuition tells me that the sample mean which is normally distributed should have the same characteristics of the actual distribution, so I try that. I draw 10,000 samples of 30 without replacement from my sample, giving me a pretty normal looking distribution with n=10000, μ=74.43, and σ=1.98. Using this in an unpaired t-test I get a p value of less than 0.0001, indicating that these are two significantly different populations. However, this seems too convenient - am I massaging the statistics here?

ANOVA with sample means

As a screening question, test users self-identified themselves according to their experience using similar tools. They could choose to answer totally inexperienced, a little experienced/ familiar, or experienced. Their distributions are like this, which it's hard to tell anything from:

I try taking means of 10000 samples of size 10 from each of the groups, giving this histogram: I want to know whether these three groups actually have different means, or if they all belong to the same group. So, I calculate the One-Way ANOVA for both the sample means and the original data:

Original data: f-measure=4.02, pvalue=0.021422.

Sample means data: f-measure=29034, pvalue=0.0.

The sample means data is a way more clear-cut case, does this mean that I've "tampered" with the data?

Can I do these operations using sample means?

Answer 1

Can I do a t-test to check whether my sample of test scores are significantly higher than that benchmark score?

Ignoring that there are really 3 sub populations within your sample of 86, yes you can. The central limit theorem provides the normality requirement for this test. I would use something like Welch's t-test and opt not to pool the variance. From a cursory look, it doesn't appear your sample and the benchmark have the same variance though I haven't computed confidence intervals for the variance estimate of your sample. In any case, assuming the variances are different and using something like Welch's test shouldn't impact power in a huge way.

am I massaging the statistics here?

It sounds like you are doing something similar to bootstrapping here but I can't be sure. There really is no need here for resampling techniques, though I suppose you could do them.

Can I do these operations using sample means?

Again, I don't think there is a need to perform resampling here. ANOVA, if I recall, is fairly robust to departures from normality so it is probably best to just use your raw data to compute the test.

Are you able to share your data and give us a little background on the problem you are trying to solve? If not, can you provide a reasonable looking fake dataset?

---

EDIT

OK, looking over your data here are my thoughts. The scores look to me to be some sort of grade out of 100. An ANOVA (which uses a gaussian likelihood) could still be used but I think it might be better to use something which respects the support of the data. Consequently, I will be using a beta regression.

Fitting the model is easy in something like R. I've compared a model with and without your group variable using the likelihood ratio test; there seems to be something in the group variable which helps explain the observed variation. The estimated means are 74.7, 70.0, and 81.0 for the little, experienced, and no groups respectively.

Here is a plot of the estimated means along with confidence intervals from the model. From the model, it looks like the means between the no group and the experienced group are significant, but the difference between little and experienced is not.