# Multivariate statistics sampling and standard error problem

I have a group of workers listening to audio from many various audio files and typing it out. I want to compare the average transcription accuracy of these old workers with 2 new workers.

I have a test dataset that I am using to get the average accuracy of the old workers and compare them with the new workers. The dataset has 40 audio files, which I have had 3 random workers type out from the old group of workers and each of the 2 new workers type out. I have accuracies for each individual case.

This leads me to the statistics questions:

• **What is the appropriate sample size for number of audio files and/or workers?
• How can I find the standard error for the average?
• I care mostly about the difference in accuracy, so how can I find the error of the difference in accuracy between the old workers and each of the
new workers?**

I have tried several different methods, but I think my statistics background is too limited to find a good solution. Here are some of the methods I have tried:

• T test – It gave me a low p value, but after thinking more about it I realized a t test may not work since the data is correlated (each datapoint for old vs new workers is them typing out the same audio file)
• Breaking standard error into three parts: Variance of accuracy between audio files, variance of workers accuracy for each audio file, and variance across a single file (accuracy can be broken down into a binomial distribution, where the worker got each word correct or incorrect, so that would give a variance of np(1-p)). Then adding these three variances together. **** The issue I see with this approach is that it seems like we have a multivariate problem, and we are assuming the variables are independent.

I have also tried several basic formulas for sample size calculation. The problems I keep running into is it seems to be a multivariate problem. The accuracy depends on the worker, and on the audio file (different files are harder to hear/type). Once I hit that problem, I struggle with finding helpful answers through google. Thoughts?

*****EDIT***** A paired t test was mentioned below as a solution, but then I have error from the average of each of the workers sample size. Is there any way to factor this into the paired t test? EDIT2 Some people have mentioned a two way ANOVA test as the best way to get an appropriate sample size. Does anyone have thoughts on this?

It seems that this is the case for a classic paired t-test.

A paired t-test is used to compare two population means where you have two samples in which observations in one sample can be paired with observations in the other sample.

In your case, you can pair the observations for the same recording and in each case you can calculate: Di = Xi - Yi

Xi = Average of the accuracy of the old workers for audio file i Yi = Average of the accuracy of the new workers for audio file i Di = Difference of average accuracy of old workers vs new workers.

Having your variables Di calculated for your 40 observations, you can calculate the Standard Deviation and average and perform a simple t-test and calculate your confidence interval.

If Your confidence interval includes 0, it means there is no statistical difference between the old and new workers accuracy.