I'm relatively new to machine learning (started about 5 months ago), and I'm looking at potentially implementing an ensemble classifier as part of my research.

I have built 3 models that I use to classify whether sales data is going to win or lose. Each model produces the probability of the sale winning or losing, and then I apply thresholds to those to classify them as either a "Win", "Loss" or "Borderline Loss". There are 25 variables, all of which are discrete.

The three models are Naive Bayes, Tree Augmented Naive Bayes (TAN) and Logistic Regression. I am using the bnlearn package for the bayesian classifiers, and a simple glm for the Logistic Regression. All models have high accuracy performances when tested on unseen data:

Naive Bayes Accuracy: 88%

TAN Accuracy: 91%

Logistic Regression Accuracy: 92%

I want to try implementing an ensemble classifier to see if I can get the best possible accuracy across all three models. My question is, how do I go about implementing something like this? I can't find too many examples online, at least not with these models for implementing one. From what I have read, one way to do it is to have a voting system, where if the 2 models predict the sale will win, but 1 predicts with will lose, then it is classified as a win. But what happens in this case if all 3 models had different predictions? I have all my prediction data ready, as in I have all the test data and each models prediction for each sale, my question so is, how would I proceed from here?

If someone knows of any available resources or tutorials that may help, I would greatly appreciate it!

  • What does TAN mean? – Sycorax Apr 7 '16 at 15:27
  • Tree Augmented Naive Bayes. Sorry I'll edit the post to make that clearer – Eoin Apr 7 '16 at 15:30
up vote 1 down vote accepted

You're discarding information by taking continuous predictions and making them into categories. I would strongly advise against taking that path. Instead, you could (1) average or (2) stack the models. Averaging is straightfoward and requires no additional tuning; on the other hand, all models are weighted equally which might be suboptimal. Stacking adds another layer of learning to train the model on the model outputs against unseen data outcomes.

  • What exactly do you mean by discarding information? All the variables I am using are categorical and discrete. I don't see how making them into categories would discard information on them? By averaging, do you mean averaging the actual accuracies? Or averaging the predictions of each individual sale opportunity? – Eoin Apr 7 '16 at 15:53
  • Instead of knowing $p(\text{win}),$ the categories only tell you if $p(\text{win})>c$ where $c$ is your threshold. The degree of correctness is important, because it captures the difference between 0.9 and 0.51. If c=0.5, then both are treated the same. – Sycorax Apr 7 '16 at 15:56
  • I'm not sure I follow what you mean. For example take the Naive Bayes classifier. I can get the associated probabilities of Win and Lose for each opportunity. So I then take that probability and apply my thresholds to them like so predictions <- ifelse(naiveprob$Won >= .8, "WIN", ifelse(naiveprob$Won >=.4, "BORDERLINE LOSS", "LOSS")) . The reason I have to do this is because at my job they have a fourth model which categorizes each opportunity into those 3 categories, and I do the same so they are easily comparable, and I can say that my new model performs better than their current one – Eoin Apr 7 '16 at 16:04
  • My overall goal would be to try and generate an ensemble classifier of the 3 models I have built, and then be able to compare the results of that ensemble to the already existing model my workplace uses. That is why I need to categorize the results into 3 categories – Eoin Apr 7 '16 at 16:05
  • Yeah, I understand what thresholding means. My point is that it's poor practice because it pretends that 0.81 and 0.99 are the same (both are "wins") even though the difference between 0.81 and 0.99 is sizable. This is an improper scoring rule, and will usually pick the wrong model. This paper provides some more context on why accuracy isn't great pages.cs.wisc.edu/~shavlik/roc.pdf It's better to use a proper scoring rule like Brier score, log-loss, c-statistic, etc depending on what your goal is. If your firm is committed to this procedure, though... who cares what best practice is... – Sycorax Apr 7 '16 at 16:09

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.