3
$\begingroup$

In making a document classifier with scikit-learn, I could easily do so with a straightforward Naive Bayes (NB) classifier like MultinomialNB.

However, I also have numeric features related to these documents, which might hold greater or equal importance to the text itself. Let's suppose that a Random Forest classifier (RF) performs well on these numeric features.

The straightforward ensembling approach would be to take the partial information available to each classifier, averaging predictions based on confidence. However, better to provide all information to RF, I believe, as follows:

I could create a pipeline that first trains NB, then feeds its class probability predictions as a feature into the RF classifier, along with all the remaining numeric features that NB never accounted for. However, the RF classifier would then overfit to the NB prediction feature, given that NB has already seen class labels at training time.

What is the canonical approach to avoiding overfitting in this two-classifier pipeline problem? My hunch is that I should be adding some amount of noise to the NB classifier's prediction probability feature that gets input into RF at training time.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

As you correctly identified this pipe-line introduces over-fitting, this over-fitting is due to data-leakage and thus renders the current form of this particular modelling pipe-line unusable.

Let's see why data-leakage happens: When feeding the class probability predictions from the NB model as a feature into the RF model, despite the fact that the NB probabilities where fitted without using features available to the RF, you pass information about the predicted labels to the RF model. This information should be unavailable to the RF at the time of prediction or training. This pipe-line is therefore flawed so no amount of noise will fix it. Unfortunately data-leakage references as sparse, "Leakage in Data Mining: Formulation, Detection, and Avoidance" by Kaufman et al. and "Ten Supplementary Analyses to Improve E-commerce Web Sites" by Kohavi and Parekh are nice reads. In general the works of S. Rosset and R. Kohavi seem to touch upon this matter more seriously.

I think you have essentially worked out a basic model-stacking procedure but unfortunately you have mixed up the dataset split. Model stacking is a staple technique for most data science competitors because it allows combining different models together; Kagglers have some very good tutorials eg. here. A canonical reference on stacking is Wolpert's "Stacked generalization"; "Combining Estimates in Regression and Classification" by LeBlanc and Tibshirani is pretty important too. In short, the class probability predictions from a base classier (the NB model here) are treated as meta-features thus allowing the top classifier (the RF model here) to use them directly as highly relevant features. Importantly, these meta-features have to derived without using information (in terms of predictors variables $X$ as well as target values/labels $y$) that will be used in the fitting procedure of the top classier. The simplest way of ensuring data-leakage will be to split the original training dataset to two training datasets; one to be used by the base and one by the top classifier. The overall performance will be estimated as always on a separate test dataset.

Improve this answer
$\endgroup$
10
  • $\begingroup$ I disagree. In particular, you write "you pass information about the predicted labels to the RF model. This information should be unavailable to the RF at the time of prediction or training." If you set up the cross-validation correctly, then at train-time, the RF will only be seeing the naive Bayes model's predictions for the same cases that it's training on. At test-time, it will only see predictions for cases that the Bayes model didn't see the true value of, either. $\endgroup$
    – Kodiologist
    Commented Apr 23, 2017 at 5:13
  • $\begingroup$ @Kodiologist: Apologies, I probably explained this inadequately. What is propose is the same thing you comment it should be done (setup CV correctly at first instance by completely separating the training sets of the two classifiers). Please check what deal leakage in stacking is. This question is a typical example, a similar but less severe example in discussed here (about 2/3 down). $\endgroup$
    – usεr11852
    Commented Apr 23, 2017 at 12:14
  • $\begingroup$ That isn't what I said should be done. I still don't see how my proposal would cause the sort of leakage effect described in the article you've linked, either. It can't be that "the meta features in {fold2, fold3, fold4, fold5} are dependent on the target values in fold1" if you're computing the "meta features" without using fold1. $\endgroup$
    – Kodiologist
    Commented Apr 23, 2017 at 14:32
  • 2
    $\begingroup$ @Kodiologist Let D be a dumb model that simply memorizes the training data completely in a single, deep decision tree. Suppose that instead of stacking RF on NB, we are stacking RF on D. Since D perfectly learns all targets in the training data, RF will find D to be a perfectly informative feature; RF has now overfit to the output of D, and the stacked model will perform miserably. $\endgroup$
    – Brian Bien
    Commented Apr 23, 2017 at 17:13
  • 1
    $\begingroup$ @BrianBien Ah, now I understand. That's a good point. The CV will still reflect the performance of the model accurately, but the performance will likely be dragged down by this internal overfitting. $\endgroup$
    – Kodiologist
    Commented Apr 23, 2017 at 18:54
1
$\begingroup$

Why use an ensemble at all in this case? A random forest can cope fine with dichotomous features, so just give the random forest all the features.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I'd like to evaluate the performance of chaining models vs simply using one or ensembling them. Also, I'm curious if there's a name for this technique (chaining?) and how others address the overfitting concern I raised. $\endgroup$
    – Brian Bien
    Commented Apr 22, 2017 at 19:57
  • $\begingroup$ @BrianBien I'm not familiar with any established term for this kind of ensemble method. To avoid overfitting, use cross-validation, only fitting both models (the naive Bayes classifier and the random forest) with the data outside the current fold, then evaluating the ensemble with the data in the fold. $\endgroup$
    – Kodiologist
    Commented Apr 22, 2017 at 20:43
  • $\begingroup$ I don't see how cross validation would solve the problem of the RF model overfitting to the output feature from the NB model. Can you elaborate on how you'd apply CV here? $\endgroup$
    – Brian Bien
    Commented Apr 22, 2017 at 21:54
  • $\begingroup$ @BrianBien The same way as you'd apply CV for any other model—where's the new danger of overfitting? You're not treating the outputs of the naive Bayes model as correct values of the DVs; you're using them as a feature. $\endgroup$
    – Kodiologist
    Commented Apr 22, 2017 at 22:43
  • 2
    $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment. $\endgroup$
    – gung - Reinstate Monica
    Commented May 6, 2017 at 2:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.