I built a model for a binary target using oversampled data. The population target prevalence is 0.25. I oversampled to 0.5 by keeping the entirety of the minority class and sampling a portion of the majority class. I then built a precision recall table using sklearn's:
#y is the binary target #PR_CV are out of bag predictions precision, recall, tr = precision_recall_curve(y, PR_CV)
I'd now want to know what precision and recall look like on the original population. I tried implementing the following, based on this article:
#odds = p/(1-p) odds = y.mean()/(1-y.mean()) print 'oversampled odds:',odds original_fraction = 0.25 original_odds = original_fraction/ (1 – original_fraction) print "original odds:",original_odds #Scoring_odds = scoring_results / (1 – scoring_results) scoring_results = PR_CV print "probability to revert:", scoring_results scoring_odds = scoring_results/(1-scoring_results) print"scoring odds:", scoring_odds #adjusted_odds = Scoring_odds * original_odds / oversampled_odds adjusted_odds = scoring_odds * original_odds / odds print "adjusted odds:", adjusted_odds #adjusted_probability = 1 / (1 + 1/adjusted_odds)) adjusted_probability = 1 / (1 + 1/adjusted_odds) print "adjusted probability:", adjusted_probability
I then calculated:
precision_adj, recall_adj, tr_adj = precision_recall_curve(y, PR_CV_adj)
What this returns is the exact same values as
That fails my intuition...What is the right way to do this?
The question I'm truing to answer is: if on an oversampled population I expect at the top 5th percentile to have x true positives over X predicted positives (precision), then on the entire population what are my x2 true positives and my X2 predicted positives?