I'm using R and running RandomForest, a GBM, and an SVM. I have prediction probabilities for all 3 models based on a binary response variable. What I want to do is run another learner on this dataset of 4 columns (the true binary response variable, and the prediction probabilities for each observation from each model as the other 3 columns) and output just one probability that would represent the best probability based on the fact that we know the truth. i.e. this way we can 'downweight' the bad predictions from one of the models while 'upweighting' the good predictions from the other models.

Is there a good method for doing this? I'm thinking about using another SVM to do this but I'm unsure how to obtain the results I'm looking for. If I run an SVM with the binary response variable and the other 3 predictors, how would I output the probability estimate I described above?

EDIT: Here is a sample dataframe in case someone wants to code in R.

df <- cbind(response = sample(rep(c(0,1), 100)), prob1 = runif(100), prob2 = runif(100), prob3 = runif(100))
  • $\begingroup$ The relevant search term is model ensemble. $\endgroup$ – Roland Dec 2 '16 at 14:21
  • $\begingroup$ And for (a lot) more information see this post by mlwave $\endgroup$ – phiver Dec 2 '16 at 14:24

This is known as Ensemble methods as mentioned above... a simple way to think about it is that you can calculate a combination of both predictions with a weight assigned to each prediction.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.