1
$\begingroup$

We know that bagging can reduce the variance of the estimator, while keep the bias around the same. If we use bagging to ensemble multiple least square regressors, then are we going to reduce the variance of the estimator as well?

If this is the case, then I am confused. We know that, based on a few assumptions, least square is the optimal estimator. How can an optimal estimator be improved?

If bagging can further improve the performance of least square regression, then, at least one of the following 2 must be true:

  1. either we changed the assumptions made by LS, so that it is no longer an optimal estimator;

  2. or, we changed the the way 'performance' is defined, it is no longer norm-2 error. For instance, the performance becomes norm-2 error PLUS a term that measures the 'robustnest' of the model (as per the following comment)

Can someone clarify for me please?

Thanks

Improve this question
$\endgroup$
1
  • $\begingroup$ Least square estimator may be optimal, but it is not very robust. Bagging would make sense also with least square regression in order to improve robustness. $\endgroup$
    – Florian
    Apr 6, 2019 at 20:04

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.