Test of significant for success rate improvement on paired samples

I want to compare the success rate of two configurations of a program. A program run two times on the same set of photographs and returns each time a list of face matches.

The success rate is computed for the two run with number_of_good_match/total_number_of_match.

e.g. Configuration A : 4500 success over 5000 matches Configuration B : 4400 success over 4900 matches

The two-proportion z-test would have been a good candidate, but it make the assumption that "The sampling method for each population is simple random sampling." which does not look apply to my data.

• What would be a good way to test that a given configuration success rate is significantly higher than another one ?
• Should I use a one-tailed test in this situation, or would a two tailed test be suitable ?

First, if you need a convenient way to test this, the pooled or un-pooled t-test might be a good fit for your problem. If you assume that Configuration A and Configuration B have the same variance, use the pooled t-test; Otherwise, use the un-pooled t-test. Both the two tests do not have restrictive assumptions on the sample size for the two groups.

Here is a resource that you can look up to for the equations: https://onlinecourses.science.psu.edu/stat200/node/60

The assumptions for the two tests are also i.i.d sampling and the underlying distribution of the two population are normally distributed. I am not sure why you think your data is not simple random sampling, but as the data size is large, CLT can always support this. Just try this pooled or the un-pooled test, and see the result if it makes any sense.

Use two tailed, if your hypothesis is group A is different from group B; Use one tailed, if your hypothesis is one group is over another.

Another test you might think of is KS test. Here is one resource: http://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm

Instead of comparing the mean of the two group, this can compare everything. For example, if the mean of two groups are very closed, but the max, min, quantile are very different for two groups, t-test might not be significant, but KS test can be significant.

One last try is Bayesian. Assuming the two groups are normally distributed, and get priors for mean and variance. Plug this in rJags. Adjust the priors for mean and the variance, see what you can get.

• I am not sure it is simple random sampling because the program runs two times on the same set of photos that I sampled for the population of all photos that exist in the world. When I renew the set of photos, I observe a change of the success rate that is two to three times higher than the difference between the change due to the program configuration. – Pierre Mar 9 '16 at 8:50
• You talk about assuming the variance are the same, but success_rate is a proportion, then the variance depends only on the parameter p. Do you think it is safe? – Pierre Mar 9 '16 at 9:30
• The first idea is if you can design an experiment? For example, resample from the population for N sets of 5000, the configuration A should apply to these N sets of data; Then you resample from from the population again for M sets of 4900 data, the configuration B will be applied to these M data sets. Then, you will have N values under Confi A and M values under Confi B, you can use the un-pooled t-test. My another idea is why you cannot just apply the Confi B on the 5000 data used in the Confi A? If you can, you can resample N sets of 5000 data, and apply both A and B on these 5000 data. – Shijia Bian Mar 9 '16 at 15:54