# How to proceed with building an ensemble classifier using Naive Bayes, TAN and Logistic Regression in R

I'm relatively new to machine learning (started about 5 months ago), and I'm looking at potentially implementing an ensemble classifier as part of my research.

I have built 3 models that I use to classify whether sales data is going to win or lose. Each model produces the probability of the sale winning or losing, and then I apply thresholds to those to classify them as either a "Win", "Loss" or "Borderline Loss". There are 25 variables, all of which are discrete.

The three models are Naive Bayes, Tree Augmented Naive Bayes (TAN) and Logistic Regression. I am using the bnlearn package for the bayesian classifiers, and a simple glm for the Logistic Regression. All models have high accuracy performances when tested on unseen data:

Naive Bayes Accuracy: 88%

TAN Accuracy: 91%

Logistic Regression Accuracy: 92%

I want to try implementing an ensemble classifier to see if I can get the best possible accuracy across all three models. My question is, how do I go about implementing something like this? I can't find too many examples online, at least not with these models for implementing one. From what I have read, one way to do it is to have a voting system, where if the 2 models predict the sale will win, but 1 predicts with will lose, then it is classified as a win. But what happens in this case if all 3 models had different predictions? I have all my prediction data ready, as in I have all the test data and each models prediction for each sale, my question so is, how would I proceed from here?

If someone knows of any available resources or tutorials that may help, I would greatly appreciate it!

• What does TAN mean? – Sycorax Apr 7 '16 at 15:27
• Tree Augmented Naive Bayes. Sorry I'll edit the post to make that clearer – Eoin Apr 7 '16 at 15:30

• Instead of knowing $p(\text{win}),$ the categories only tell you if $p(\text{win})>c$ where $c$ is your threshold. The degree of correctness is important, because it captures the difference between 0.9 and 0.51. If c=0.5, then both are treated the same. – Sycorax Apr 7 '16 at 15:56
• I'm not sure I follow what you mean. For example take the Naive Bayes classifier. I can get the associated probabilities of Win and Lose for each opportunity. So I then take that probability and apply my thresholds to them like so predictions <- ifelse(naiveprob$Won >= .8, "WIN", ifelse(naiveprob$Won >=.4, "BORDERLINE LOSS", "LOSS")) . The reason I have to do this is because at my job they have a fourth model which categorizes each opportunity into those 3 categories, and I do the same so they are easily comparable, and I can say that my new model performs better than their current one – Eoin Apr 7 '16 at 16:04