I have a group of workers listening to audio from many various audio files and typing it out. I want to compare the average transcription accuracy of these old workers with 2 new workers.
I have a test dataset that I am using to get the average accuracy of the old workers and compare them with the new workers. The dataset has 40 audio files, which I have had 3 random workers type out from the old group of workers and each of the 2 new workers type out. I have accuracies for each individual case.
This leads me to the statistics questions:
- **What is the appropriate sample size for number of audio files and/or workers?
- How can I find the standard error for the average?
- I care mostly about the difference in accuracy, so how can I find the error
of the difference in accuracy between the old workers and each of the
I have tried several different methods, but I think my statistics background is too limited to find a good solution. Here are some of the methods I have tried:
- T test – It gave me a low p value, but after thinking more about it I realized a t test may not work since the data is correlated (each datapoint for old vs new workers is them typing out the same audio file)
- Breaking standard error into three parts: Variance of accuracy between audio files, variance of workers accuracy for each audio file, and variance across a single file (accuracy can be broken down into a binomial distribution, where the worker got each word correct or incorrect, so that would give a variance of np(1-p)). Then adding these three variances together. **** The issue I see with this approach is that it seems like we have a multivariate problem, and we are assuming the variables are independent.
I have also tried several basic formulas for sample size calculation. The problems I keep running into is it seems to be a multivariate problem. The accuracy depends on the worker, and on the audio file (different files are harder to hear/type). Once I hit that problem, I struggle with finding helpful answers through google. Thoughts?
*****EDIT***** A paired t test was mentioned below as a solution, but then I have error from the average of each of the workers sample size. Is there any way to factor this into the paired t test? EDIT2 Some people have mentioned a two way ANOVA test as the best way to get an appropriate sample size. Does anyone have thoughts on this?