T-test with duplicate instances

Question

Suppose we are sampling N = 7 queries out of all queries issued to a search engine A. Since some queries are more popular than others, we end up with U = 3 unique queries in our sample. Assume the search engine A always returns the same documents. Hence, all instances of the same unique query gets the same relevance score.

query           sample_weight   score_a   score_b
==============  =============   =======   =======
nba                   4           0.7       0.8
stimulus check        2           0.6       0.6
hamster               1           0.9       0.1

We want to compare the effectiveness of our control search engine with a new search engine B on this same set of queries. The difference in the means on this sample can be compared in either of the following equivalent manners:

• [4 * (0.8 - 0.7) + 2 * (0.6 - 0.6) + 1 * (0.1 - 0.9)] / 7
• [(0.8 - 0.7) + (0.8 - 0.7) + (0.8 - 0.7) + (0.8 - 0.7) + (0.6 - 0.6) + (0.6 - 0.6) + (0.1 - 0.9)] / 7

Now, what gets me confused is how to do a paired t-test. Should I do it based on the unique queries (i.e., with two vectors of size 3), or on all occurrences (i.e., with two vectors of size 7). In other words, which of the following is correct?

1. paired_t_test([0.7, 0.6, 0.9], [0.8, 0.6, 0.1])
2. paired_t_test([0.7, 0.7, 0.7, 0.7, 0.6, 0.6, 0.9], [0.8, 0.8, 0.8, 0.8, 0.6, 0.6, 0.1])
3. Some thing else

Answer 1

If literally all you were going to do is run the t-test and choose one of the systems to use, go with the 7. (Or, in the real case, the 7 million or whatever.)

But almost certainly you want to do a lot more than that (including special handling of heavy hitter queries), and there's 20 questions I would ask before we even got to the choice of a statistical test. My real answer is get Ronny Kohavi's book:

https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264