6
$\begingroup$

Colleagues of mine recently presented a work where they calibrate boosted regression trees (BRT) models on small data sets ($n= 30$). They validated the models using leave-one-out cross validation (LOOCV) using R2, RMSPE and RPD indices. They also provided these indices computed by training and validating the model on the full dataset. The R2, RMSPE and RPD values obtained through LOOCV were almost strictly equal to the R2, RMSPE and RPD values obtained when validating on the training data set.

My questions are :

  • Is such a results expected for LOOCV on BRT?

  • Is this because BRT is relatively insensitive to outliers (and to single individuals?) that excluding one individual during LOOCV does not make a difference, providing nearly similar calibrated models with same performance metrics on the excluded individuals?

  • In that case does LOOCV for BRT makes any sense, compared to repeated k-fold CV with $k < n$?

Thank you in advance

$\endgroup$
4
$\begingroup$

It is hard to tell without data, but the set may be "too homogeneous" to make LOO work -- imagine you have a set $X$ and you duplicate all objects to make a set $X_d$ -- while BRT usually have very good accuracy on its train, it is pretty obvious that LOO on $X_d$ will probably give identical results to test-on-train.

So if the accuracy if good I would even try resampling CV (on each of let's say 10 folds you make train of an equal size to the full set by sampling objects with replacement and test from objects that were not placed in train -- this should spit them in about 1:2 proportion) on this data to verify this result.

EDIT: More precise algorithm of resampling CV

Given a dataset with $N$ objects and $M$ attributes:

  1. Training set is made by randomly selecting $N$ objects from the original set with replacement
  2. The objects that were not selected in the step 1 form the test set (this is roughly $\frac{1}{3}N$ objects)
  3. Classifier is trained on a train set and tested on test set, and the measured error is gathered
  4. Steps 1-3 are repeated $T$ times, where $T$ is more less arbitrary, say 10, 15 or 30
$\endgroup$
  • $\begingroup$ @mbq: Can you explain this idea of re-sampling CV? Are you proposing an algorithm where you create 10 groups (typical 10 fold CV) but within each fold you create 1 bootstrap sample and determine the expected error on those observations not in the training set of the bootstrap? So you still have 10 error measurements to average? $\endgroup$ – B_Miner Dec 29 '10 at 18:42
  • $\begingroup$ @mbq: Also, Are you suggesting this (compared to say the regular 10 fold cross validation) because of the small sample size? $\endgroup$ – B_Miner Dec 29 '10 at 18:46
  • $\begingroup$ @user2040 I can't find a good reference, so I'll edit the answer to be more precise; due to the second thing, yes, I think OP needs a quite small N (to remove this possible homogeneity bias) for N-fold CV, and so may get too few repetitions for a meaningful result. $\endgroup$ – user88 Dec 29 '10 at 21:09
  • $\begingroup$ @mbq: Thank you. From what I read of your edit, this is the usual bootstrap validation (e.g. Elements of Statistical Learning 2nd edition page 249), but with 10,15 or 30 iterations (compared to 100's or 1000's) typically used with large N data set. $\endgroup$ – B_Miner Dec 30 '10 at 0:35
  • $\begingroup$ Thanks for the replies. The (small) dataset is not that homogeneous : the coefficient of variation reaches 26% for the modelled property they are modelling using 3 predictors. That is why I was looking more at the robustness of BRTs to the inclusion or exclusion of very few individuals (1 in case of LOOCV), and thus, more generally speaking the validity of LOOCV for BRTs. $\endgroup$ – backflip Dec 30 '10 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.