0
$\begingroup$

I have a group of workers listening to audio from many various audio files and typing it out. I want to compare the average transcription accuracy of these old workers with 2 new workers.

I have a test dataset that I am using to get the average accuracy of the old workers and compare them with the new workers. The dataset has 40 audio files, which I have had 3 random workers type out from the old group of workers and each of the 2 new workers type out. I have accuracies for each individual case.

This leads me to the statistics questions:

  • **What is the appropriate sample size for number of audio files and/or workers?
  • How can I find the standard error for the average?
  • I care mostly about the difference in accuracy, so how can I find the error of the difference in accuracy between the old workers and each of the
    new workers?**

I have tried several different methods, but I think my statistics background is too limited to find a good solution. Here are some of the methods I have tried:

  • T test – It gave me a low p value, but after thinking more about it I realized a t test may not work since the data is correlated (each datapoint for old vs new workers is them typing out the same audio file)
  • Breaking standard error into three parts: Variance of accuracy between audio files, variance of workers accuracy for each audio file, and variance across a single file (accuracy can be broken down into a binomial distribution, where the worker got each word correct or incorrect, so that would give a variance of np(1-p)). Then adding these three variances together. **** The issue I see with this approach is that it seems like we have a multivariate problem, and we are assuming the variables are independent.

I have also tried several basic formulas for sample size calculation. The problems I keep running into is it seems to be a multivariate problem. The accuracy depends on the worker, and on the audio file (different files are harder to hear/type). Once I hit that problem, I struggle with finding helpful answers through google. Thoughts?

*****EDIT***** A paired t test was mentioned below as a solution, but then I have error from the average of each of the workers sample size. Is there any way to factor this into the paired t test? EDIT2 Some people have mentioned a two way ANOVA test as the best way to get an appropriate sample size. Does anyone have thoughts on this?

$\endgroup$
0
$\begingroup$

It seems that this is the case for a classic paired t-test.

A paired t-test is used to compare two population means where you have two samples in which observations in one sample can be paired with observations in the other sample.

In your case, you can pair the observations for the same recording and in each case you can calculate: Di = Xi - Yi

Xi = Average of the accuracy of the old workers for audio file i Yi = Average of the accuracy of the new workers for audio file i Di = Difference of average accuracy of old workers vs new workers.

Having your variables Di calculated for your 40 observations, you can calculate the Standard Deviation and average and perform a simple t-test and calculate your confidence interval.

If Your confidence interval includes 0, it means there is no statistical difference between the old and new workers accuracy.

You can read more about paired t-test here

You can use this calculator to calculate the confidence interval between

$\endgroup$
  • $\begingroup$ Thanks for your response! This seems to fit the problem pretty well. Do you know if there is there a certain number of workers I need to get Xi? It seems like I am adding a little bit of error there by averaging only 3 out of the pool of workers. Is there some way to calculate this error? $\endgroup$ – pastahacker Aug 24 '18 at 23:17
  • $\begingroup$ The sample size you need to prove your hypothesis depends on the average D and the standard deviation of the difference. It's not the same for every problem. Since you are measuring D you probably don't know it, but you can decide what is a minimal effect you want to be able to measure. Let's say you want to be able to detect a difference of at least 5% accuracy, you might need N observations, but to be able to prove a 1% accuracy you might need M observations (M>N). There are sample size calculators that might help you like this sample-size.net/sample-size-study-paired-t-test $\endgroup$ – Ary Jazz Aug 25 '18 at 7:03
  • $\begingroup$ I'm a little confused. I understand that the sample size of D depends on the standard deviation of Di. But, for each data point I will have 3 points from the old workers (I can get more, but I don't know how many to use for the sample), and 1 point from one of the new workers. So, should I average the three points from the old workers and then get Di using that average? If so, then I need to make sure I have enough points to have an accurate Xi, right? I should have some sort of error on Xi, how do I propagate that through the paired samples? $\endgroup$ – pastahacker Aug 27 '18 at 15:52
  • $\begingroup$ Or should I be calculating Di by (for each audio recording) subtracting the new worker accuracy from each of the three samples of the old worker accuracy and treating each of those as separate datapoints? $\endgroup$ – pastahacker Aug 27 '18 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.