Combining machine learning models

Question

I'm kind of new to datamining/machine learning/etc. and have been reading about a couple ways to combine multiple models and runs of the same model to improve predictions.

My impression from reading a couple papers (which are often interesting and great on theory and greek letters but short on code and actual examples) is that it's supposed to go like this:

I take a model (knn, RF, etc) and get a list of classifiers between 0 and 1. My question is how to do combine each of these lists of classifiers? Do I run the same models on my training set so that the number of columns going into the final model are the same or is there some other trick?

It would be great if any suggestions/examples included R code.

NOTE: This is for a data set w/ 100k lines in the training set and 70k in the test set and 10 columns.

Answer 1

It actually boils down to one of the "3B" techniques: bagging, boosting or blending.

In bagging, you train a lot of classifiers on different subsets of object and combine answers by average for regression and voting for classification (there are some other options for more complex situations, but I'll skip it). Vote proportion/variance can be interpreted as error approximation since the individual classifiers are usually considered independent. RF is in fact a bagging ensemble.

Boosting is a wider family of methods, however their main point is that you build next classifier on the residuals of the former, this way (in theory) gradually increasing accuracy by highlighting more and more subtle interactions. The predictions are thus usually combined by summing them up, something like calculating a value of a function in x by summing values of its Taylor series' elements for x.
Most popular versions are (Stochastic) Gradient Boosting (with nice mathematical foundation) and AdaBoost (well known, in fact a specific case of GB). From a holistic perspective, decision tree is a boosting of trivial pivot classifiers.

Blending is an idea of nesting classifiers, i.e. running one classifier on an information system made of predictions of other classifiers. As so, it is a very variable method and certainly not a defined algorithm; may require a lot of objects (in most cases the "blender" classifier must be trained on a set of objects which were not used to build the partial classifiers to avoid embarrassing overfit).
The predictions of partial classifiers are obviously combined by melding them into an information system which is predicted by the blender.

Answer 2

The expression "combining models" is vague, but my guess is that you are asking about ensemble learning methods. The best reference to learn about them is perhaps Rich Caruana's papers:

http://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml04.icdm06long.pdf

There is no actual code in this paper, but the algorithm is clearly described, so you shouldn't have any problem coding it in any language you prefer.

Answer 3

After looking around for a bit with the help of the above answers, I realized what my problem was. I had been trying to use the predictions of other models in the same model as was used to predict them. In other words if I had 5 variables when I ran the kNN model, I'd add a new variable with the predictions of the kNN model and make a model with 6 variables when I ran the random Forest model. I found that instead the results from the models should be segmented and run as a separate model. So I'd create predictions from various models (knn, RF, svd, etc.) then run a separate combining/blending/stacking model using just the predictions as variables and the classifications as what to predict.

I think some of my problems were that in combining the predictions with the other variables, there may have been some overfitting or multicollinearity but I'm not certain. Someone else might be able to weigh in better on what I was doing wrong. Anyways thanks to everyone for their help.

Answer 4

Good publication for combining multiple predictors is: Taraba (2023), "Optimal blending of multiple independent prediction models", Front. Artif. Intell., 6.

It contains formulas for combining the models based on their variances as well as formula for the variance of the final blend. It also provides upper estimate based on maximum variance of all models. All with proofs :) \begin{equation} {y}_{B,j} = \sum_{k=1}^{N} \hat{\alpha_k} {y}_{k,j} \end{equation} Having $N$ independent models with normal distributions $R_{k} \sim \mathcal{N}(0,\sigma_k^2)$ for $k \in [1,N]$, we get an optimal blend with parameters \begin{equation} \hat{\alpha_k} = \frac{\prod_{\substack{j=1 \\ j \ne k}}^N \sigma_j^2}{\sum_{i=1}^{N} \prod_{\substack{j=1 \\ j \ne i}}^N \sigma_j^2}, \end{equation} and these independent models form normal distribution $\mathcal{N}(0,\sigma_B^2)$, which has variance \begin{equation} \sigma_B^2 = \frac{\prod_{j=1}^N \sigma_j^2}{\sum_{i=1}^{N} \prod_{\substack{j=1 \\ j \ne i}}^N \sigma_j^2}. \end{equation} Having $N$ independent models with normal distributions $R_{k} \sim \mathcal{N}(0,\sigma_k^2)$ for $k \in [1,N]$ and their variances $\sigma_k^2 \leq \sigma_M^2$, where $\sigma_M^2$ is their maximum variance, combining them optimally with coefficients from previous formula, their combined variance is $\sigma_B^2 \leq \frac{\sigma_M^2}{N}$.