4
$\begingroup$

In collective learning (ensemble methods) we need the estimators to be Independent/ uncorrelated from one another. Do I understand correctly, that this means we need to draw the data samples without replacement?

$\endgroup$

1 Answer 1

3
$\begingroup$

No, the independence of estimators does not come from the sampling technique.

Empirically, ensembles tend to achieve better performance when there is more diversity between models. To achieve this diversity (not to be confused with independence), randomized training sets can be generated (with or without replacement). Models are then trained on different samples (see e.g. Bootstrap Sampling/Bagging) and then combined e.g. by averaging. Parallel training is possible because your estimators are independent.

In contrast, boosting is an example of an ensemble method based on sequential training where each estimator tries to correct the mistakes of the previous basic learner. By definition, learners are not independent.

$\endgroup$
4
  • $\begingroup$ Ah I see ! So when we say the estimators are independent, it means they do not learn iteratively ("through past experience"). A model that does not learn through iterations would be bagging. But if we have a model that learns sequentially, like Boosting we have dependent estimators? $\endgroup$
    – DataBach
    Commented Dec 11, 2019 at 15:01
  • 1
    $\begingroup$ first of all it means that training one estimator does not affect the training process of another. "through past experience" does not matter here. $\endgroup$ Commented Dec 11, 2019 at 15:11
  • 1
    $\begingroup$ Yes to the bagging part. Training Boosting algorithms cannot be parallelized, because estimators are dependent, not the other way around. $\endgroup$ Commented Dec 11, 2019 at 15:14
  • 1
    $\begingroup$ Ok. Thank you for helping me clarify! $\endgroup$
    – DataBach
    Commented Dec 11, 2019 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.