How to Avoid Overfitting in Spam Classification with Text and Numeric Features In making a document classifier with scikit-learn, I could easily do so with a straightforward Naive Bayes (NB) classifier like MultinomialNB.
However, I also have numeric features related to these documents, which might hold greater or equal importance to the text itself. Let's suppose that a Random Forest classifier (RF) performs well on these numeric features.
The straightforward ensembling approach would be to take the partial information available to each classifier, averaging predictions based on confidence. However, better to provide all information to RF, I believe, as follows:
I could create a pipeline that first trains NB, then feeds its class probability predictions as a feature into the RF classifier, along with all the remaining numeric features that NB never accounted for. However, the RF classifier would then overfit to the NB prediction feature, given that NB has already seen class labels at training time.
What is the canonical approach to avoiding overfitting in this two-classifier pipeline problem? My hunch is that I should be adding some amount of noise to the NB classifier's prediction probability feature that gets input into RF at training time.
 A: As you correctly identified this pipe-line introduces over-fitting, this over-fitting is due to data-leakage and thus renders the current form of this particular modelling pipe-line unusable.
Let's see why data-leakage happens: When feeding the class probability predictions from the NB model as a feature into the RF model, despite the fact that the NB probabilities where fitted without using features available to the RF, you pass information about the predicted labels to the RF model. This information should be unavailable to the RF at the time of prediction or training. This pipe-line is therefore flawed so no amount of noise will fix it. Unfortunately data-leakage references as sparse, "Leakage in Data Mining: Formulation, Detection, and Avoidance" by Kaufman et al. and "Ten Supplementary Analyses to Improve E-commerce Web Sites" by Kohavi and Parekh are nice reads. In general the works of S. Rosset and R. Kohavi seem to touch upon this matter more seriously.
I think you have essentially worked out a basic model-stacking procedure but unfortunately you have mixed up the dataset split. Model stacking is a staple technique for most data science competitors because it allows combining different models together; Kagglers have some very good tutorials eg. here. A canonical reference on stacking is Wolpert's "Stacked generalization"; "Combining Estimates in Regression and Classification" by LeBlanc and Tibshirani is pretty important too. In short, the class probability predictions from a base classier (the NB model here) are treated as meta-features thus allowing the top classifier (the RF model here) to use them directly as highly relevant features. Importantly, these meta-features have to derived without using information (in terms of predictors variables $X$ as well as target values/labels $y$) that will be used in the fitting procedure of the top classier. The simplest way of ensuring data-leakage will be to split the original training dataset to two training datasets; one to be used by the base and one by the top classifier. The overall performance will be estimated as always on a separate test dataset.
A: Why use an ensemble at all in this case? A random forest can cope fine with dichotomous features, so just give the random forest all the features.
