0
$\begingroup$

I am creating species distribution models using the popular R package . I am using functionality to create ensemble models. Ensemble models combine individual models based on their weight of evidence (statistical performance).

Traditionally, model averaging or multi-model inference was performed using likelihood techniques (i.e., AIC). More recently, techniques have been developed to average models that are not likelihood based, such as Random Forests. This is accomplished by using cross-validation or split sample validation to assess statistical performance and then combine models using measures of statistical performance like ROC, kappa, TSS, etc.

My question is how can individual models be assessed for statistical accuracy using cross-validation or split sample, then be combined into an ensemble weighted based on performance, and then the ensemble model be assessed for accuracy using cross-validation or split sample again with independent evaluation data? The issue here is that when you create different partitions of calibration and evaluation data to assess the individual models, there ends up not being any independent data left to evaluate the ensemble model.

The only way that I can think that this would work is to use cross-validation and ensemble within runs. Each run of cross-validation would have the same partition of data for all models (i.e., if you were doing 5-fold cross validation on 3 models, in Run 1 the three models would use the same 20% of data as evaluation and 80% as calibration, in Run 2 the three models would use the same 20% of data as evaluation and 80% as calibration, etc.). That way, all three models in each run can be combined, and the resulting ensemble model could be assessed using the same 20% of evaluation data that was used for the individual models.

The issue with this arrangement is that it results in 5 separate models, one for each run of the cross-validation. That isn't a very satisfying result.

Any recommendations for best practices on creating ensemble models and then evaluating them? I realize people are probably going to say a dataset that is truly independent in space and time that is saved for assessment of the final ensemble model. Any alternatives to this for people with data that isn't independent in space and time and small dataset that aren't big enough to have a portion left out for a final evaluation?

$\endgroup$

1 Answer 1

0
$\begingroup$

Ensemble learning works as follows:

For each candidate estimator obtain the cross-validated predictions. That is: construct 10 disjoint validation sets, train the estimator 10 times on the training sets, obtain all validation predictions, stack the validation predictions together to obtain a prediction vector for all samples.

Next, one can plug the cross-validated prediction vector of each estimator into risk functions (e.g. least-squares risk) to get an unbiased estimate of performance. Choosing the best one would be cross-validation selection.

Alternatively, one can look at every linear/convex/nonnegative combination of the cross-validated prediction vectors across the estimators. For each combination, you can plug it in the least-squares risk function to obtain an unbiased risk estimate. You can then choose the best linear/convex/nonnegative combination based on these risk estimates. This is equivalent to fitting linear regression/nonnegative least-squares with the cross-validated predictions as covariates/features. This is called stacking.

To evaluate the ensemble learner, one should perform nested/double cross-validation to get a fair risk estimate of the ensemble/stacked learner itself and then compare this risk estimate to the risk estimates of the individual learners/estimators. In other words, treat the ensemble learner as an estimator in-and-of-itself and use (nested) cross-validation to compare it to the non-stacked learners. If you do 10-fold cross-validation then you should have enough data to do the nested cross-validation. Alternatively, you can increase the folds to ~15-20 if you have very little data. Also, using hold-out data to evaluate the ensemble model is a very inefficient method.

For references, google "SuperLearner"

$\endgroup$
2
  • $\begingroup$ I have attempted to give you reputation points. Once I reach 15 point you will receive those. The methodology you outlined is very informative. I am going to wait to accept the answer for 1-2 days in hopes of hearing from someone from the biology/ecology field that might have an answer more closely related to species distribution modeling and biomod2, as the methodology you outlined is quite different than what is typically done in this field. $\endgroup$
    – nateroe
    Aug 9, 2021 at 21:42
  • $\begingroup$ No need for that. I am just glad this was helpful! $\endgroup$
    – user327671
    Aug 10, 2021 at 3:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.