I'm trying to get a handle on what the current state of things is when it comes to quantifying uncertainty in nonparametric regression models. It seems like the options are

  • Use a Bayesian model and hope the model fits well enough to be able to trust the posterior predictive distributions it comes up with
  • Use a non-Bayesian model for which there are known uncertainty quantification techniques, e.g. random forest + jackknife
  • Use a generally applicable trick that can be applied to any model, e.g. bootstrap confidence intervals, conformal prediction
  • Try to model the residuals of your main model so that you can work out something like the "expected error" of the main model on a new datum

I have a few questions:

  1. Have I overlooked any potential approaches in the above list?
  2. Are there other good, flexible regression methods that people have figured out uncertainty quantification for? Random forests are the only such method I've come across.
  3. Are people using bootstrap confidence intervals/conformal prediction on large datasets in production? I haven't managed to find much on this.

I realise that this is a broad question but hope that's ok. Thanks!

  • $\begingroup$ What do you mean by uncertainty? Confidence bounds? $\endgroup$ – Karel Macek Jan 17 '18 at 7:00
  • $\begingroup$ @KarelMacek yeah anything that can give an idea of how much to "trust" a prediction really, confidence bounds would be good, doesn't necessarily have to be completely statistically sound $\endgroup$ – Taimur Jan 17 '18 at 11:40
  • $\begingroup$ For Bayesian regression models, you can use alternative R^2 proposed by Andrew Gelman et al, stat.columbia.edu/~gelman/research/unpublished/bayes_R2.pdf $\endgroup$ – cwl Jan 17 '18 at 12:52
  • $\begingroup$ In addition, posterior predictive check (PPC) also provides a way to check how well your model can reproduce the data. $\endgroup$ – cwl Jan 17 '18 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.