2
$\begingroup$

In ensemble learning, we average the predictions of multiple base learners (e.g. SVM + ANN + Linear regression). Instead of taking the mean of the individual base models' predictions, can lasso be used somehow to intelligently weight the individual predictions, in other words, decide that the ensemble prediction should be 45% SVM, 40% ANN and 15% linear regression? Lasso is normally used for feature selection, but isn't applying it for model selection also a technique? Source papers would be nice.

and if not lasso, how about using an optimization algorithm like particle swarm to intelligently weight the base learners' predictions instead?

$\endgroup$
2
$\begingroup$

LASSO is indeed used for computing weights of forecast combinations; see Diebold & Shin (2019). The twist is that forecast weights are shrunk towards $1/k$ (where $k$ is the number of forecasts being averaged) rather than $0$ as LASSO normally would.

The rationale for shrinking the combination weights towards $1/k$ is the empirical fact that equally-weighted forecast combinations tend to outperform unequally-weighted forecast combinations, known as the forecast combination puzzle. The reason behind the latter is estimation imprecision (due to high variance of estimators) of theoretically optimal weights; see Claeskens et al. (2016) or Diebold (2017) Chapter 12, among other studies. LASSO allows reducing estimation variance at a cost of increasing bias. As long as the variance reduction is greater than the increase in the square of the bias (which it typically is for suitably selected penalization intensity), the mean squared error is reduced (hint: bias-variance decomposition and trade-off).

References

$\endgroup$
1
$\begingroup$

There is also the concept of Stacked Regression. Given a set of predictions $v$ from different models, you can apply constrained least squares in order to find a weighted average combination that minimizes prediction error. It can be formulated as: $min \sum_n(y_{i}-\sum_ka_kv_{kn})^2, s.t. a_k\geq0, \sum a_k=1$. The paper also provides a theoretical discussion on why constraint least squares outperforms ridge regression, although it would not make a huge difference to apply unconstrained least squares.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.