I have a prototype machine producing parts.

In a first test the machine produces $N_1$ parts and a binary classifier tells me that $d_1$ parts are defective ($d_1 < N_1$, usually $d_1/N_1<0.01$ and $N_1\approx10^4$) and $N_1-d_1$ parts are good.

Then a technician makes some changes in the machine in order to decrease the number of defective parts.

In a second and following test the modified machine produces $N_2$ parts and the same binary classifier (untouched) tells me that $d_2$ parts are defective, anyway $d_2/N_2$ is quite similar to $d_1/N_1$.

The technician would like to know if his changes are effective.

Assuming that the classifiers is perfect (its sensitivity is 100% and its specificity is 100%), I can perform a test for proportions (with R, I just type prop.test(c(d1,d2),c(N1,N2))).

But the classifier is not perfect, so how can I take in account the sensitivity and the specificity, both unknown, of the classifier in order to properly answer to the technician?


So I'm deriving this from first principles, and am thus not certain it is correct. Here's my thoughts:

EDIT: This was not quite right before. I have updated it.

  1. Let's let $\alpha$ denote the expected difference between the actual number of true positives $d_1$ and the number output by the binary classifier which we'll call $\hat{d_1}$. You can measure this with by running your classifier on a set with known labels. Subtract the number of actual positives from the number of positives produced by the classifier, and then divide by $N$ to get $\alpha$.

  2. So, a point estimate for the actual ratio of defective parts is given by: $\hat{\frac{d_1}{N_1}} = \frac{d_1 + \alpha * N_1}{N_1}$. That is, the observed number of defective parts, less the expected number of false positives, plus the expected number of false negatives.

  3. Similarly, $\hat{\frac{d_2}{N_2}} = \frac{d_2 + \alpha * N_2}{N_2}$

  4. So, now let's do a prop test. In the standard prop test, we first compute the pooled ratio used as the null value: $p= \frac{p_1*N_1 + p_2*N_2}{N_1 + N_2}$. So here, we put in our point estimates of $\hat{\frac{d_1}{N_1}}$ and $\hat{\frac{d_2}{N_2}}$ to get: $p= \frac{d_1 + d_2 + \alpha * (N_1 + N_2)}{N_1 + N_2}$

  5. And then standard error is just the usual: $\sqrt{p*(1-p)*(\frac{1}{N_1} + \frac{1}{N_2})}$

  6. And the test statistic is the same: $z = \frac{\frac{d_1}{N_1} - \frac{d_2}{N_2}}{se}$

Some thoughts on interpretation:

  • The model can produce imaginary values for standard error. This will happen when $p < 0$, which will be the case when the number of errors we expect the classifiers to produce exceeds the number we observed. For example, suppose that we expect our classifier to produce an average of 5 positives even when given a sample containing no positives. If we observe 4 positives, then it's as though there is no signal: Our result is indistinguishable from the noise produced by the classifier. In this case, we should not reject the null hypothesis, I think.

  • Another way to think about this is that, if the number of defective parts is within the margin of error for the classifier then of course we cannot tell whether there is a difference: we can't even tell whether any parts are defective!

Incorporating errors in the estimation of $\alpha$:

  • I thought about this some more, and I think there are several ways you could do this, but essentially you want to get an estimate of the distribution of $\alpha$. Ideally you would do this buy repeating your procedure for getting the estimate of $\alpha$ on a representative sample of the data sets you intend to use this method on. If this is not possible, you could bootstrap on a single dataset by drawing samples from it, although this is not ideal unless your single datset is representative of all the sets you care about.

Suppose that we want to compute a confidence interval with a confidence of $h$.

  • Empirically compute the $\frac{h}{2}$ confidence interval over $\alpha$ using the bootstrapped distribution. Plug each end point into the process above, using it as a (very conservative or very liberal) point estimate for $\alpha$ and find the $\frac{h}{2}$ confidence interval for the estimate of the difference in proportions using the prop test. Suppose that we get intervals ($low_l, low_r)$ and $(high_l, high_r)$ as the intervals for the lower and higher values of $\alpha$. Then the interval $(high_l,low_r)$ (which contains both of the earlier intervals) should be a (1-h)*100 % CI for the difference in proportions... I think...

Note: In the above I assumed a 1 sided test. You divide h by 2 to account for the fact that you are testing two independent hypotheses ($\alpha$ being in the interval you think and the test statistic being a significant difference). If you want to do a two-tailed test, divide by 4 instead.

| cite | improve this answer | |
  • $\begingroup$ +1, thank you. In 6 you wrote "static", did you mean "statistic"? $\endgroup$ – Alessandro Jacopson Oct 3 '12 at 16:56
  • $\begingroup$ In your first bullet point you consider $p<0$ giving an imaginary standard error. What about $0<p<1$ (which can give an imaginary standard error too)? Is it possible to get $0<p<1$ ? $\endgroup$ – Alessandro Jacopson Oct 3 '12 at 17:40
  • $\begingroup$ In your second bullett point, you wrote about the "variance", what do you mean? My understanding is the following: let's say I take a sample of size $0.01(N1−d1)\approx100$ from the good ones of the first test and found 7 defective parts, then if I assume $\beta=\frac{7}{100}$ I will ignore any variance in the $\beta$. On the other hand I can get a confidence interval for $\beta$ (for example with R prop.test(7,100)) and then incorporate it in the model. Am I right? $\endgroup$ – Alessandro Jacopson Oct 3 '12 at 17:59
  • $\begingroup$ @uvts_cvs Yep, that should be "statistic". I'll fix it in a moment. There's also a typo in the calculation for standard error, which should be p*(1-p) instead. P should always be < 1, except maybe if your classifier is really bad and d is large. For your third comment, yeah, that's the idea. I'm just not sure how to incorporate that estimate into the model. Perhaps someone else here knows? $\endgroup$ – John Doucette Oct 3 '12 at 20:22
  • $\begingroup$ Thanks for accepting, but since last night I've thought some more on it (very good question by the way!), and have some ideas on how to incorporate the variance in. Also, I realized that this model is not quite right. Need to multiply $\alpha$ by the number of negative exemplars and $\beta$ by the number of positive exemplars. I'll work it through and update this later on. $\endgroup$ – John Doucette Oct 4 '12 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.