I have a prototype machine producing parts.
In a first test the machine produces $N_1$ parts and a binary classifier tells me that $d_1$ parts are defective ($d_1 < N_1$, usually $d_1/N_1<0.01$ and $N_1\approx10^4$) and $N_1-d_1$ parts are good.
Then a technician makes some changes in the machine in order to decrease the number of defective parts.
In a second and following test the modified machine produces $N_2$ parts and the same binary classifier (untouched) tells me that $d_2$ parts are defective, anyway $d_2/N_2$ is quite similar to $d_1/N_1$.
The technician would like to know if his changes are effective.
Assuming that the classifiers is perfect (its sensitivity is 100% and its specificity is 100%), I can perform a test for proportions (with R, I just type
But the classifier is not perfect, so how can I take in account the sensitivity and the specificity, both unknown, of the classifier in order to properly answer to the technician?