Test for proportions and binary classifier I have a prototype machine producing parts.
In a first test the machine produces $N_1$ parts and a binary classifier tells me that $d_1$ parts are defective ($d_1 < N_1$, usually $d_1/N_1<0.01$ and $N_1\approx10^4$) and $N_1-d_1$ parts are good.
Then a technician makes some changes in the machine in order to decrease the number of defective parts.
In a second and following test the modified machine produces $N_2$ parts and the same binary classifier (untouched) tells me that $d_2$ parts are defective, anyway $d_2/N_2$ is quite similar to $d_1/N_1$.
The technician would like to know if his changes are effective.
Assuming that the classifiers is perfect (its sensitivity is 100% and its specificity is 100%), I can perform a test for proportions (with R, I just type prop.test(c(d1,d2),c(N1,N2))).
But the classifier is not perfect, so how can I take in account the sensitivity and the specificity, both unknown, of the classifier in order to properly answer to the technician?
 A: So I'm deriving this from first principles, and am thus not certain it is correct. Here's my thoughts:
EDIT: This was not quite right before. I have updated it.


*

*Let's let $\alpha$ denote the expected difference between the actual number of true positives $d_1$ and the number output by the binary classifier which we'll call $\hat{d_1}$. You can measure this with by running your classifier on a set with known labels. Subtract the number of actual positives from the number of positives produced by the classifier, and then divide by $N$ to get $\alpha$.

*So, a point estimate for the actual ratio of defective parts is given by: 
$\hat{\frac{d_1}{N_1}} = \frac{d_1 + \alpha * N_1}{N_1}$. That is, the observed number of defective parts, less the expected number of false positives, plus the expected number of false negatives.

*Similarly, $\hat{\frac{d_2}{N_2}} = \frac{d_2 + \alpha * N_2}{N_2}$

*So, now let's do a prop test. In the standard prop test, we first compute the pooled ratio used as the null value: $p= \frac{p_1*N_1 + p_2*N_2}{N_1 + N_2}$. So here, we put in our point estimates of $\hat{\frac{d_1}{N_1}}$ and $\hat{\frac{d_2}{N_2}}$ to get: $p= \frac{d_1 + d_2 + \alpha * (N_1 + N_2)}{N_1 + N_2}$

*And then standard error is just the usual: $\sqrt{p*(1-p)*(\frac{1}{N_1} + \frac{1}{N_2})}$

*And the test statistic is the same: $z = \frac{\frac{d_1}{N_1} - \frac{d_2}{N_2}}{se}$
Some thoughts on interpretation: 


*

*The model can produce imaginary values for standard error. This will happen when $p < 0$, which will be the case when the number of errors we expect the classifiers to produce exceeds the number we observed. For example, suppose that we expect our classifier to produce an average of 5 positives even when given a sample containing no positives. If we observe 4 positives, then it's as though there is no signal: Our result is indistinguishable from the noise produced by the classifier. In this case, we should not reject the null hypothesis, I think. 

*Another way to think about this is that, if the number of defective parts is within the margin of error for the classifier then of course we cannot tell whether there is a difference: we can't even tell whether any parts are defective! 
Incorporating errors in the estimation of $\alpha$:


*

*I thought about this some more, and I think there are several ways you could do this, but essentially you want to get an estimate of the distribution of $\alpha$. Ideally you would do this buy repeating your procedure for getting the estimate of $\alpha$ on a representative sample of the data sets you intend to use this method on. If this is not possible, you could bootstrap on a single dataset by drawing samples from it, although this is not ideal unless your single datset is representative of all the sets you care about.


Suppose that we want to compute a confidence interval with a confidence of $h$. 


*

*Empirically compute the $\frac{h}{2}$ confidence interval over $\alpha$ using the bootstrapped distribution. Plug each end point into the process above, using it as a (very conservative or very liberal) point estimate for $\alpha$ and find the $\frac{h}{2}$ confidence interval for the estimate of the difference in proportions using the prop test. Suppose that we get intervals ($low_l, low_r)$ and $(high_l, high_r)$ as the intervals for the lower and higher values of $\alpha$. Then the interval $(high_l,low_r)$ (which contains both of the earlier intervals) should be a (1-h)*100 % CI for the difference in proportions... I think...


Note: In the above I assumed a 1 sided test. You divide h by 2 to account for the fact that you are testing two independent hypotheses ($\alpha$ being in the interval you think and the test statistic being a significant difference). If you want to do a two-tailed test, divide by 4 instead.
