Lasso for Ensemble Learning, base learner selection

Question

In ensemble learning, we average the predictions of multiple base learners (e.g. SVM + ANN + Linear regression). Instead of taking the mean of the individual base models' predictions, can lasso be used somehow to intelligently weight the individual predictions, in other words, decide that the ensemble prediction should be 45% SVM, 40% ANN and 15% linear regression? Lasso is normally used for feature selection, but isn't applying it for model selection also a technique? Source papers would be nice.

and if not lasso, how about using an optimization algorithm like particle swarm to intelligently weight the base learners' predictions instead?

Answer 1

LASSO is indeed used for computing weights of forecast combinations; see Diebold & Shin (2019). The twist is that forecast weights are shrunk towards $1/k$ (where $k$ is the number of forecasts being averaged) rather than $0$ as LASSO normally would.

The rationale for shrinking the combination weights towards $1/k$ is the empirical fact that equally-weighted forecast combinations tend to outperform unequally-weighted forecast combinations, known as the forecast combination puzzle. The reason behind the latter is estimation imprecision (due to high variance of estimators) of theoretically optimal weights; see Claeskens et al. (2016) or Diebold (2017) Chapter 12, among other studies. LASSO allows reducing estimation variance at a cost of increasing bias. As long as the variance reduction is greater than the increase in the square of the bias (which it typically is for suitably selected penalization intensity), the mean squared error is reduced (hint: bias-variance decomposition and trade-off).

References

• Claeskens et al. "The forecast combination puzzle: A simple theoretical explanation" (2016)
• Diebold "Forecasting in Economics, Business, Finance and Beyond" (2017)
• Diebold & Shin "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives" (2019)

Answer 2

There is also the concept of Stacked Regression. Given a set of predictions $v$ from different models, you can apply constrained least squares in order to find a weighted average combination that minimizes prediction error. It can be formulated as: $min \sum_n(y_{i}-\sum_ka_kv_{kn})^2, s.t. a_k\geq0, \sum a_k=1$. The paper also provides a theoretical discussion on why constraint least squares outperforms ridge regression, although it would not make a huge difference to apply unconstrained least squares.