I have a dataset of about 1M observation and I had to predict a response that occurs only about 10.000 times (1%).

I decided to train a random forest, but this takes a lot of time to train because the data is too large for my hardware. So I decided to take a sample, but an aleatory sample would be already too large to have a minimum quantity of response. (If I take 10% aleatory, i would have only 1000 response)

Then I took a stratified sample. All responses and 10.000 aleatory non-responses, and trained my model in this dataset.

But now I need to rescale the probability so I have the real probability of the observation to be response.

I tried to simulate this problem with this code in R. Training a model in a balanced dataset and another one in the unbalanced data. But those models are not very correlated and I didn't find a good way to tranform the probabilities to the original unbalanced scale.

enter image description here

I found that for logistic regression I can do this by just changing the intercept this way:

$$ \hat{\beta_0} = \hat{\beta_0^*} - log(\frac{\gamma_1}{\gamma_2})$$

Where $ \gamma_1 = Pr(Z=1|Y=1)$ and $ \gamma_2 = Pr(Z=1|Y=0)$. $Z$ is the an aleatory variable indicating if the observation is in the reduced dataset.

This was found in this book (in portuguese) page 216

simulate_data <- function(n){
  X <- data.frame(matrix(runif(n*20), ncol = 20))
    X = X,
    Y = rbinom(n, size = 1, prob = apply(X, 1, sum) %>% pnorm(mean = 13)) %>% as.factor()  

balance <- function(X, Y){
  X <- rbind(
    X[Y == "1",],
    X[Y == "0",] %>% sample_n(length(Y[Y == "1"]))
    X = X,
    Y = as.factor(c(rep(c(1,0), each = length(Y[Y == "1"]))))

train <- simulate_data(100000)

m_desb <- mean(train$Y == "1")
modelo_desb <- randomForest(train$X,train$Y, ntree = 200, cutoff = c(1-m_desb, m_desb), nodesize = 30, mtry = 8)
bal <- balance(train$X, train$Y)
m_bal <- mean(bal$Y == "1")
modelo_bal <- randomForest(bal$X,bal$Y, ntree = 100, cutoff = c(1-m_bal, m_bal), nodesize = 50)

Code for plot

  unbalanced = predict(modelo_desb, newdata = test$X, type = "prob")[,2],
  balanced = predict(modelo_bal, newdata = test$X, type = "prob")[,2]
) %>%
  ggplot(aes(x = balanced, y = unbalanced)) +
  geom_point(size = 0.3) +
  xlim(0,1) +
  geom_smooth() +
  geom_hline(yintercept = m_desb, linetype = "dashed") +
  geom_vline(xintercept = m_bal, linetype = "dashed")

If your trained model is (very) good, the stratification should be no problem, because the forest classify the observations correctly in spite of the stratification.

What is your hardware? If you have several cpus you can use the R-package ranger, this is probably much faster and yields the same results as randomForest.

You can also train random forests iteratively on parts of the datasets and at the end put them together. 1 M is not too much, depending on your number and type of features you can easily train a random forest model with 50000 observations

There are also other packages like h2o, ... And you can use AWS Amazon, ...

Maybe a simpler model than random forest (e.g. simply rule based) but trained on the whole dataset can also provide better results, than using only a part of the data.

edit: What you can do, is also to use the strata and sampsize argument (as a vector) in the randomForest package. There you can specify exactly the number of samples you want to have from each of the classes. And in different trees you can get completely different samples as you sample from all observations.

| cite | improve this answer | |
  • $\begingroup$ Classifications are much less useful than probabilities but a particular problem with classifications is they do not transport to other settings with much different base probabilities. Classifications constructed from a sample with one prevalence are not relevant to a sample with a much different prevalence; the classifications will be far from optimal. A great advantage of logistic regression is that you only need to modify the intercept. The modification may not be as simple as what is described above. $\endgroup$ – Frank Harrell Jul 17 '16 at 11:31
  • $\begingroup$ I have no problems with my model trained in the balanced dataset. My problem is that I need to know the true probability of response for each observation, not the probability given the observation is in the training dataset (what my model is answering). @FrankHarrell by the way the classification using the model trained in the balanced dataset was better then the one trained in the unbalanced one. (If you run the simulation code you'll see this) $\endgroup$ – Daniel Falbel Jul 18 '16 at 19:23
  • $\begingroup$ That's because you are using an improper accuracy scoring rule. This misleads. $\endgroup$ – Frank Harrell Jul 19 '16 at 3:20
  • $\begingroup$ @PhilippPro could you expand a little on how to use the strata and sampsize arguments? Also, do you have any suggestions on how to do this without having to train my model again. (I'm doing a one vs. rest classifier so I have a lot of models to train..) $\endgroup$ – Daniel Falbel Jul 28 '16 at 14:44
  • $\begingroup$ Minimal example: library(randomForest) model = randomForest(Species ~ ., data = iris, sampsize = c(40,10,10), strata = iris$Species) See also here: stackoverflow.com/questions/14842059/… This happens while training, so I think you have to train your model again. I do not understand what you mean with 1 vs. rest here. In your description it seems binary. ;) $\endgroup$ – PhilippPro Jul 29 '16 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.