1
$\begingroup$

So basically I have a machine learning model where I want to have a prediction interval, the model is XGBoost so it is tricky to do Quantile Regression and I was looking for an alternative method to achieve this goal.

I have came across this article that suggests to estimate the standard deviation for the prediction by creating another machine learning model to predict the error of the original machine learning model. So the process would be:

  1. Fit machine learning model to training data.
  2. Calculate the error for each datapoint.
  3. Fit a second machine learning model to predict the squared error of the datapoints.
  4. When making a prediction use the initial prediction as mean and the square root of the prediction of the squared error as an estimate of the standard deviation.
  5. With the mean and std deviation build a confidence interval for the prediction.

The assumptions here is that the distribution of the predictions is Normal and that we can estimate the mean and standard deviation in the way mentioned.

My question is, does this method make sense? And is there a better way to do this?

$\endgroup$
0
$\begingroup$

While I cannot fully evaluate the approach cited in the blog post you pointed to, I can at least propose another way of obtaining confidence intervals, which is via bootstrapping.

Bootstrap methods are convenient mainly for two reasons: they're simple to implement and doen't need assumption of the underlying statistics you are bootstrapping.

The implementation follows, in general, the following pseudocode:

statistics = []
for i in bootstraps:
    train, test = select_sample_with_replacement(data, size)
    model = train_model(train)
    stat = evaluate_model(test)
    statistics.append(stat)

which comes from this blog post, that has a really nice intro presentation on how to use bootstrapping to evaluate ML models.

Also check this really nice answer explaining why bootstrap works.

$\endgroup$
  • $\begingroup$ Thanks Lucas, I've actually looked into bootstrap but it's applicable only to the evuation error. Im trying to predict a prediction confidence interval. $\endgroup$ – Franco Piccolo Mar 12 at 17:12
  • $\begingroup$ @FrancoPiccolo Not at all, that's not the only case bootstrap can be used. Look here, here and here, for example. $\endgroup$ – Lucas Farias Mar 12 at 17:18
  • $\begingroup$ Thanks for the links, but I didn't find anything to work for my case where I want prediction interval for XGBoost. The links there work for linear regression. $\endgroup$ – Franco Piccolo Mar 13 at 5:08
  • $\begingroup$ @FrancoPiccolo have you seen this article? $\endgroup$ – Lucas Farias Mar 13 at 6:17
  • $\begingroup$ Yes that's what I was trying to avoid for complexity.. Maybe Ill need to go deeper there or just use a model to estimate error as in the question. $\endgroup$ – Franco Piccolo Mar 13 at 6:19
0
$\begingroup$

I'm looking at a similar question and found this blog is useful: https://medium.com/@qucit/a-simple-technique-to-estimate-prediction-intervals-for-any-regression-model-2dd73f630bcb

$\endgroup$
  • $\begingroup$ This is not an appropriate use of the answer box. $\endgroup$ – Michael Chernick Oct 31 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.