# Determine if an action increases the proportion of 1s in binary data with unknown population

We're trying to improve search results, and we're trying to determine if certain changes (adding a word to a synonym list, removing it from the query, etc) have a statistically significant improvement on the results for future searches.

We currently have data for past queries and their matching "correct" results. We also have tests to sample from those past queries, perform them again, and look at the percentage of "good matches". A good match being one for which the known correct result is within the top recommendations by the search engine.

For a concrete example: say we want to know if adding a synonym for show->display will improve results. We look through past queries and find all of those containing the word 'show', then we find the number of good matches. Then we add the synonym map, and test again for the same sample of queries.

The results let us know how many queries improved, how many got worse, how many stayed the same, total number of samples, and total number of good matches (before and after the test).

To get an idea of the numbers, we have about 260,000 past queries, and for most words there are less than 10,000 queries which contain those words (and thus we can test the entire sample).

For this example the results are:

Sample size containing 'show': 5250
Good matches (base):           3212
Good matches (test):           3208
Improved:                      8
Got worse:                     12
Total change:                  -4

The question is which test(s) would be the most helpful in determining if this change is likely to represent the change in future unknown queries? (Using a 95% confidence level)

Here's the things I've tried on R, and their results:

• Two sample T-test on both sets of data (represented by vectors of 1s and 0s) - not significant
• T-test on the difference of the sets - not significant
• Bootstrapping on the difference in mean of each set, and doing a T-test of that result - significant
• Bootstrapping each sample on their mean, then T-test two sample t-test of the results - significant
• Bootstrapping each sample on their mean, then t-test of the difference in the results - significant
• McNemar test - significant
• Are you sure that McNemar's test is significant? 8 improved vs 12 worsened seems pretty balanced to me. Jan 22, 2014 at 22:03
• @Aniko well, R gives me a p-value of almost 0, not sure if I'm missing something there although I doubt it (just takes a 2x2 matrix) Jan 22, 2014 at 22:18
• @Glen_b sorry, not the best with wording.. I've changed the title to match Jan 22, 2014 at 22:19

Since you have paired data, you should be using a paired test. Even with bootstrapping, you have to make sure that you keep the pairing. With binary data, a t-test is really not the best choice. It is reasonable with a big sample size, but there are better targeted tests for binary data. So the test of choice is McNemar's test which is targeted exactly at your situation (paired binary data).

For McNemar's test you have to remember to set up the data correctly. The total should be the number of pairs (5250), and each cell should be one of the Base X / Test Y combinations. Based on your summary, I back-calculated the contents of the table:

> (mm <-  matrix(c(2030, 8, 12, 3200), nr=2))
[,1] [,2]
[1,] 2030   12
[2,]    8 3200

In mm the rows are Base bad, Base good, and the columns are Test bad, Test good. So 2030 is the number of both bad combinations, 3200 is the number of both good combinations, and 12 and 8 are the number of discrepant results.

> mcnemar.test(mm)

McNemar's Chi-squared test with continuity correction

data:  mm
McNemar's chi-squared = 0.45, df = 1, p-value = 0.5023
• Ohhh.... for some reason I had the bottom two backwards from each other. Now that I see your example I see where I messed up. So McNemar's is good for telling me if the change itself is significant based on the sample. Could I then extrapolate that it'd be significant in my unknown population if we assume that the distribution is similar? Jan 23, 2014 at 18:36
• Thinking a bit more about it, would it be reasonable to then bootstrap (paired of course) using the p-value from McNemar's test as my statistic to estimate the p-value of the population? Jan 23, 2014 at 19:22
• I think you are missing the point of a hypothesis test. It does not tell you whether there is a difference in the sample. In this case, there clearly is, because $8 \neq 12$. Rather it tries to tell you something about the population from which the sample is drawn. Jan 23, 2014 at 19:25
• Oh yeah... I get fuzzy with stats when I think too much about it. Jan 23, 2014 at 20:13