I built a model for a binary target using oversampled data. The population target prevalence is 0.25. I oversampled to 0.5 by keeping the entirety of the minority class and sampling a portion of the majority class. I then built a precision recall table using sklearn's:
#y is the binary target
#PR_CV are out of bag predictions
precision, recall, tr = precision_recall_curve(y, PR_CV)
I'd now want to know what precision and recall look like on the original population. I tried implementing the following, based on this article:
#odds = p/(1-p)
odds = y.mean()/(1-y.mean())
print 'oversampled odds:',odds
original_fraction = 0.25
original_odds = original_fraction/ (1 – original_fraction)
print "original odds:",original_odds
#Scoring_odds = scoring_results / (1 – scoring_results)
scoring_results = PR_CV
print "probability to revert:", scoring_results
scoring_odds = scoring_results/(1-scoring_results)
print"scoring odds:", scoring_odds
#adjusted_odds = Scoring_odds * original_odds / oversampled_odds
adjusted_odds = scoring_odds * original_odds / odds
print "adjusted odds:", adjusted_odds
#adjusted_probability = 1 / (1 + 1/adjusted_odds))
adjusted_probability = 1 / (1 + 1/adjusted_odds)
print "adjusted probability:", adjusted_probability
I then calculated:
precision_adj, recall_adj, tr_adj = precision_recall_curve(y, PR_CV_adj)
What this returns is the exact same values as precision_recall_curve(y, PR_CV)
That fails my intuition...What is the right way to do this?
The question I'm truing to answer is: if on an oversampled population I expect at the top 5th percentile to have x true positives over X predicted positives (precision), then on the entire population what are my x2 true positives and my X2 predicted positives?