2
$\begingroup$

I built a model for a binary target using oversampled data. The population target prevalence is 0.25. I oversampled to 0.5 by keeping the entirety of the minority class and sampling a portion of the majority class. I then built a precision recall table using sklearn's:

#y is the binary target
#PR_CV are out of bag predictions
precision, recall, tr = precision_recall_curve(y, PR_CV)

I'd now want to know what precision and recall look like on the original population. I tried implementing the following, based on this article:

#odds = p/(1-p)
odds = y.mean()/(1-y.mean())
print 'oversampled odds:',odds

original_fraction = 0.25
original_odds = original_fraction/ (1 – original_fraction)
print "original odds:",original_odds

#Scoring_odds = scoring_results / (1 – scoring_results)
scoring_results = PR_CV
print "probability to revert:", scoring_results
scoring_odds = scoring_results/(1-scoring_results)
print"scoring odds:", scoring_odds

#adjusted_odds = Scoring_odds * original_odds / oversampled_odds 
adjusted_odds = scoring_odds * original_odds / odds
print "adjusted odds:", adjusted_odds

#adjusted_probability = 1 / (1 + 1/adjusted_odds))
adjusted_probability = 1 / (1 + 1/adjusted_odds)
print "adjusted probability:", adjusted_probability

I then calculated:

precision_adj, recall_adj, tr_adj = precision_recall_curve(y, PR_CV_adj)

What this returns is the exact same values as precision_recall_curve(y, PR_CV)

That fails my intuition...What is the right way to do this?

The question I'm truing to answer is: if on an oversampled population I expect at the top 5th percentile to have x true positives over X predicted positives (precision), then on the entire population what are my x2 true positives and my X2 predicted positives?

$\endgroup$
2

2 Answers 2

1
$\begingroup$

Because the adjustment is a monotonic function, every threshold in the model's probability space has a corresponding one in the adjusted space (indeed, just apply the same adjustment) at which the confusion matrix of the resulting hard classifiers are the same. So the precision-recall curves are the same, it's just that every point corresponds to a different threshold.

$\endgroup$
0
$\begingroup$

To estimate efficacy of using sampling techniques you should first split your data between train/test set, apply sampling on training data and then estimates results on test set. The way you are doing you can leakage information and get unrealistic performances results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.