2
$\begingroup$

Suppose we are sampling N = 7 queries out of all queries issued to a search engine A. Since some queries are more popular than others, we end up with U = 3 unique queries in our sample. Assume the search engine A always returns the same documents. Hence, all instances of the same unique query gets the same relevance score.

query           sample_weight   score_a   score_b
==============  =============   =======   =======
nba                   4           0.7       0.8
stimulus check        2           0.6       0.6
hamster               1           0.9       0.1

We want to compare the effectiveness of our control search engine with a new search engine B on this same set of queries. The difference in the means on this sample can be compared in either of the following equivalent manners:

  • [4 * (0.8 - 0.7) + 2 * (0.6 - 0.6) + 1 * (0.1 - 0.9)] / 7
  • [(0.8 - 0.7) + (0.8 - 0.7) + (0.8 - 0.7) + (0.8 - 0.7) + (0.6 - 0.6) + (0.6 - 0.6) + (0.1 - 0.9)] / 7

Now, what gets me confused is how to do a paired t-test. Should I do it based on the unique queries (i.e., with two vectors of size 3), or on all occurrences (i.e., with two vectors of size 7). In other words, which of the following is correct?

  1. paired_t_test([0.7, 0.6, 0.9], [0.8, 0.6, 0.1])
  2. paired_t_test([0.7, 0.7, 0.7, 0.7, 0.6, 0.6, 0.9], [0.8, 0.8, 0.8, 0.8, 0.6, 0.6, 0.1])
  3. Some thing else
$\endgroup$

1 Answer 1

1
$\begingroup$

If literally all you were going to do is run the t-test and choose one of the systems to use, go with the 7. (Or, in the real case, the 7 million or whatever.)

But almost certainly you want to do a lot more than that (including special handling of heavy hitter queries), and there's 20 questions I would ask before we even got to the choice of a statistical test. My real answer is get Ronny Kohavi's book:

https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264

$\endgroup$
2
  • $\begingroup$ Great -- and thanks for the heads-up @Dave! $\endgroup$
    – mossaab
    Sep 2, 2022 at 0:57
  • $\begingroup$ I'm revisiting this issue and was wondering whether the degrees of freedom should be based on the count of unique queries (3) or the sum of the weights (7). $\endgroup$
    – mossaab
    Jan 16, 2023 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.