0
$\begingroup$

I have trouble of deciding whether I should go for paired or unpaired test in my case. Let's say that I need to evaluate the performance of two search engine configurations and check whether the performance of Configuration 2 is better than Configuration 1.

I can either:

  • compare the search performance on two independent queries samples and test whether the mean(metric for config 1) > mean(metric for config 2) or
  • create one sample of queries, pass them through two configuration and compare the means of metrics as well

Is it actually true that I can approach this problem in two different ways ? If that's the case, what are the consequences of going for one option over the other ?

I'd be grateful for any guidance as to how to approach it

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The paired design has greater statistical power, and should almost always be preferred where feasible. Unpaired designs are often necessary when collecting paired data would be difficult (e.g. it's too much of a time commitment for participants), but it's hard to imagine that's the case here.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the answer. As far as I see, the paired test also requires smaller minimal sample for a test to be reliable. So am I correct in saying that in order to test my hypothesis correctly, it is more a matter of creating representative sample of queries rather than creating big enough one ? $\endgroup$
    – deonardo_licaprio
    Commented Oct 13, 2022 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.