2
$\begingroup$

I have an expensive model (or class of models). My baseline approach to quantify uncertainty re the model parameters are hessian based standard errors, and I use k-fold cross validation for model comparison / validation. While a full bootstrap would be pleasant as a more robust uncertainty quantification, this is quite expensive. I think I should also be able to develop expectations for the variance of the leave-k out estimates, to at least get a rough sense of where the hessian based standard error estimates are not performing well. I wonder if someone knows how to do this, or can point to work that does this? Something like an approximate jackknife?

Improve this question
$\endgroup$
6
  • $\begingroup$ Just to be clear, was it a repeated CV or a single repeat? For example, under $K$=3 (~33% test set) we effectively have something that approximates the test set size of a standard bootstrap (~36.6%), if we did say 20x3-fold that is not miles away from 60 bootstrap samples. $\endgroup$
    – usεr11852
    May 21, 2020 at 17:08
  • $\begingroup$ The example that led to the question was a single 10 fold -- this is flexible though, to an extent. $\endgroup$
    – Charlie
    May 21, 2020 at 17:27
  • $\begingroup$ OK, scratch that then. :) Have you consider using Delta method? $\endgroup$
    – usεr11852
    May 21, 2020 at 17:29
  • $\begingroup$ That is how the hessian is approximated, yeah. I'm hoping to use the k-fold estimates as a check on this. Increasing the number of folds / repeats is certainly possible, it's just not obvious to me how to use the k-folds to approximate the std error of the parameter estimate. $\endgroup$
    – Charlie
    May 21, 2020 at 17:36
  • $\begingroup$ Good choice! What model are we talking about here? (I think it would helpful to make it clear you are already using the Delta method because it couldn't tell immediately from the original post.) Also, you can increase the folds/repeats why not just bootstrap things? My original comment was along the lines of reusing the estimates through bias correction (See Vanwinckelen & Blockeel (2012) and the CV thread here) for a more context.) $\endgroup$
    – usεr11852
    May 21, 2020 at 17:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.