As a simple example, say you want to predict the house price. You have 5 features. You build 5 models, each trained with one feature.

(price, sqft) (price, num_bedrooms) (price, lot_size) (price, num_bathrooms) (price, crime_rate)

For a test instance, you can just average the price predictions. That's one way to combine the models. Any other ways to combine models (perhaps a weighted average)?

If we average the results, how does will the results compare to just a similar model using all five features? Or how does building a bunch of simple models and combining them compare to building one complex model?

One thought is that perhaps this approach can be used when missing data is common. For example, just average the result of three models when sqft and bedrooms are missing for a test instance.

The other idea is to use the price outputs as features of a more complex model.

  • 2
    $\begingroup$ Why don't you try? $\endgroup$ – theGD Jun 4 '16 at 18:34
  • $\begingroup$ @Gun of course I could. But I was sure other had to have done something like this. I had assumed boosting used different subsets of the training data and wasn't sure which method used different subsets of the raw inputs to derive features. $\endgroup$ – yalis Jun 5 '16 at 16:37

You can absolutely combine simple models into a more complex model. This class of techniques is called ensemble methods, and can be very successful. As to whether a specific form of this idea will help on a specific problem, you'll just have to try it and see. If the models are linear, and combined linearly, then it won't help.

Some techniques to look up are bagging and boosting. These use different combinations/weightings of the data points to generate the different models, so they're slightly different than your question about using subsets of features. But, looking at the way the different models are combined and the principles behind it can give you some insight. Random forests use a combination of decision trees, where each tree is trained using a different subset of data points and features. Stacking uses a final 'meta model' to combine the predictions of lower-level models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.