In making a document classifier with scikit-learn, I could easily do so with a straightforward Naive Bayes (NB) classifier like
However, I also have numeric features related to these documents, which might hold greater or equal importance to the text itself. Let's suppose that a Random Forest classifier (RF) performs well on these numeric features.
The straightforward ensembling approach would be to take the partial information available to each classifier, averaging predictions based on confidence. However, better to provide all information to RF, I believe, as follows:
I could create a pipeline that first trains NB, then feeds its class probability predictions as a feature into the RF classifier, along with all the remaining numeric features that NB never accounted for. However, the RF classifier would then overfit to the NB prediction feature, given that NB has already seen class labels at training time.
What is the canonical approach to avoiding overfitting in this two-classifier pipeline problem? My hunch is that I should be adding some amount of noise to the NB classifier's prediction probability feature that gets input into RF at training time.