1
$\begingroup$

I have the following experimental design: I have different (and disjoint) sets of data, and for every set of input data $s_1 ... s_n$ I generate the outputs $y_1 ... y_n$ and $z_1 ... z_n$ using two different algorithms $Y$ and $Z$.

Then I present each result pair $i$ in a side-by-side ($y_i \leftrightarrow z_i$, randomized) manner to $k$ human testers who provide me with a $score \in [-10;10]$ which side was better in their opinion:

sample side-by-side experiment design

At the end of the experiment I have $k \times n$ groups of scores in the range $[-10;10]$ per set.

What would be the best approach for significance testing (e.g. to test whether $Y$ is statistically different from $Z$ with given threshold $p$ and if not what would be the required sample size $k$)?

What would be the way to compare two different sets?

In the current design every tester provides the score for every pair. I want to limit the max amount of scores per tester per set to $m, m < n$, so that I only get $\frac{k \times m}{n}$ scores per pair. What tests could I apply in this case?

$\endgroup$
1
$\begingroup$

a non-parametric T test like the Mann-Whitney U is probably what you are looking for. Even though you have enumerated your scores on a -2 to 2 scale, they are not really the mathematical distance from each other that you ascribed. In general, you arbitrarily gave them values to scores which cant be quantified. I.e., its difficult to quantify the mathematical distance between "same" and "left better".

$\endgroup$
  • $\begingroup$ Fully agree with your concern, the mapping from the scale to values is purely arbitrary and does not represent the actual distance in mathematical sense! We can, however, always flatten the results to {-1,0,1} set. Would that help? $\endgroup$ – Alexander Galkin Oct 10 '17 at 13:26
  • $\begingroup$ rescaling the data would not help with the underlying concern. What has to be done is to use a non-parametric test (i.e. a test that does not have as many assumptions or any assumptions) to assess the two groups. What this means is that you will have to sacrifice some statistical power. Non-parametric tests need larger sample sizes to reliably detect similar alpha level thresholds of parametric tests. $\endgroup$ – JWH2006 Oct 10 '17 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.