2
$\begingroup$

Let's say I have a large, un-balanced binary classification problem (in reality nrow is more like 500k, and ncol is more like 500):

set.seed(42)
nrow <- 10000
ncol <- 50
X <- matrix(rnorm(nrow*ncol), ncol=ncol)
Y <- X %*% rnorm(ncol(X)) * sample(0:1, ncol(X), replace=TRUE) + rnorm(nrow(X))
Y <- Y-20
Y <- exp(Y)/(1+exp(Y))
Y <- round(Y, 0)
> sum(Y==1)/length(Y)
[1] 0.0027

Before modeling, I down-sampled the negative class. I don't have a strong theoretical justification for doing this, but it makes my models fit a lot faster, and they seem to be better too.

keep <- which(Y==1)
sample <- sample(which(Y==0), length(keep))
Xfull <- X
Yfull <- Y
X <- X[c(keep, sample),]
Y <- Y[c(keep, sample),]
> sum(Y==1)/length(Y)
[1] 0.5

Fitting a model to the down-sampled dataset is pretty quick:

library(caret)
Y <- factor(paste('X', Y, sep=''))
X <- as.data.frame(X)
model <- train(X, Y, method='glmnet', 
               tuneGrid=expand.grid(.alpha=0:1, .lambda=0:30/10),
               trControl=trainControl(
                 method='cv', 
                 summaryFunction=twoClassSummary, 
                 classProbs=TRUE))
plot(model)

And I can use the cross-validation folds to estimate some statistics about the model's predictive ability:

> max(model$results$ROC)
[1] 0.9777778
> confusionMatrix(model)

Cross-Validated (10 fold) Confusion Matrix 

(entries are percentages of table totals)

          Reference
Prediction   X0   X1
        X0 41.0  3.3
        X1  9.0 46.7

However, I would like to estimate these statistics on the FULL dataset, preferably without cross-validating my model on the full dataset, which would be extremely slow.

I was thinking of doing a naive re-scaling of the confusion matrix, like this:

scaling_factor <- 0.5/0.0027
CM <- confusionMatrix(model)$table * nrow(X)
CM[,1] <- CM[,1]*scaling_factor
> round(CM/sum(CM)*100, 2)
          Reference
Prediction    X0    X1
        X0 81.56  0.04
        X1 17.90  0.50

Does this seem like a reasonable calculation? Is there a similar method I could use to re-scale AUC? Or do I expect AUC to stay the same?

/edit: in response to B_Miner. I am fairly certain that fitting the downsampled model to the full dataset will overestimate its performance. It's easy to see why if we fit a random forest instead of a glmnet:

model <- train(X, Y, method='rf', 
                   trControl=trainControl(
                     method='cv', 
                     summaryFunction=twoClassSummary, 
                     classProbs=TRUE))

And predict this model on the full dataset:

pred_full <- predict(model, Xfull, type='raw')
> table(pred_full, Yfull)
         Yfull
pred_full    0    1
       X0 8531    0
       X1 1442   27

Because every single positive instance was used to train the model, the model can perfectly predict these instances, even on the full dataset.

/edit2: To clarify. I understand the the down-sampled model is biased. However, I suspect that the model's bias is predictable and consistent. I'm looking for a theoretical way to correct for this bias, under the assumption that the removed negative observations come from the same distribution as the negative observations in the training set.

$\endgroup$
  • $\begingroup$ Why do you care about the performance on the training data - if you have a test set (needed if not doing CV) then why not use that (which presumably is not oversampled)? $\endgroup$ – B_Miner Apr 18 '13 at 17:11
  • 1
    $\begingroup$ Why not use the model obtained to predict on the full dataset and obtain the confusion matrix from that? $\endgroup$ – Affine Apr 18 '13 at 17:13
  • $\begingroup$ @Affine, also makes sense. $\endgroup$ – B_Miner Apr 18 '13 at 17:17
  • $\begingroup$ @B_Miner Even predicting on a test set is extremely slow for the full dataset. I'd really like use the down-sampled estimates if there's something valid I can do with them. $\endgroup$ – Zach Apr 18 '13 at 17:29
  • $\begingroup$ @Affine: Because my model uses ALL of the positive class, predicting on the full dataset will probably lead to over-fitting. Think of a random forest-- if you run a random forest's training data back through the model, you tend to get perfect predictions. $\endgroup$ – Zach Apr 18 '13 at 17:30
3
$\begingroup$

In response to the comments, here is the general answer on how to adjust the probabilities returned out of any predictive model that has been built on a stratified / oversampled data set. Since you artificially increased the density of one class, you need to adjust the predicted probability that the observation belongs to that class in the real world space.

Do note that this only works if the probability returned is indeed a probability of class membership and is well calibrated in the oversampled space. If for example, you are returning confidence values from an SVM, then this rescaling does not work and you need to first calibrate the scores (e.g. Platt).

If you have a model calculating probability of class membership where those predicted probabilities are close to the actual (on the oversampled data) probabilities (I normally decile the data and compare actual and predicted at each decile) then you can adjust as follows (I forget the source for this but comes from Bayes Theorem and is used by SAS EM for example). Original fraction is the proportion of "1s" in the full data, Oversampled fraction is the proportion of "1s" in the oversampled training set and scoring result is the probability from the model.

enter image description here

$\endgroup$
  • $\begingroup$ This might be a silly question, but in the denominatior you have 1 + (....-1). Don't the 1's cancel out? $\endgroup$ – Zach Apr 18 '13 at 20:57
  • $\begingroup$ Note the bracket. It does not cancel out. $\endgroup$ – B_Miner Apr 18 '13 at 21:09
  • $\begingroup$ Here is the same formula in SAS code. Note &var is a macro variable. p_adj=1/(1+(1/%sysevalf(&orig_den)-1)/(1/%sysevalf(&mod_den)-1)*(1/score-1)); $\endgroup$ – B_Miner Apr 18 '13 at 21:11
  • 1
    $\begingroup$ I see. There should be another bracket around (1/score-1) $\endgroup$ – Zach Apr 18 '13 at 21:19
  • $\begingroup$ I found the site the image came from: data-mining-blog.com/tips-and-tutorials/… note in the comments someone else asked about canceling and the author claimed the image was correct. $\endgroup$ – B_Miner Apr 18 '13 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.