2
$\begingroup$

I am trying to simulate predictions of a binary classifier, given its precision, recall, classification accuracy ($CA$) and the actual classes of the instances.

I.e. I have instances and I know what their actual classes are. I'm trying to create errors in those classes that would accurately reflect the error rates I would've received from the classifier with the above properties.

I would like someone to double-check my process:

Call the classes 1 (positive) and 0 (negative). The following is the information that's available:

$$ Precision = Pr[class = 1\ |\ prediction = 1]\\ Recall = Pr[prediction = 1\ |\ class = 1]\\ CA = Pr[prediction = 1\ \&\ class=1] + Pr[prediction = 0\ \&\ class = 0] $$

Recall gives us the probability that the prediction is 1 if the true class is 1. Therefore I should change 1s to 0s with probability equal to $1 - Recall$.

Now we need to calculate $NPV = Pr[prediction = 0\ |\ class = 0]$. The basic formula of conditional probability gives us: $$ Recall = \frac{Pr[prediction = 1\ \&\ class = 1]}{Pr[class = 1]}\\ NPV = \frac{Pr[class = 0\ \&\ prediction = 0]}{Pr[class = 0]} $$

Continuing from above for $NPV$: $$ NPV=\frac{CA - Pr[prediction = 1\ \&\ class = 1]}{Pr[class = 0]} =\\ =\frac{CA - Recall \cdot Pr[class = 1]}{Pr[class = 0]} $$

Since I know the values of $Pr[class = 0]$ and $Pr[class = 1]$, I can now calculate $NPV$ and change 0s to 1s in my data with probability $1-NPV$.

Is this correct?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.