I have a regression model (not necessarily linear regression) and a test set. I would like to be able to say: "If you use my model, then with probability 95%, the prediction error will be in the interval [a,b]" where the prediction error is the difference between predicted y and the real y as given by the test data.

Is there any method to estimate the values a and b from the test set?

I know that using bootstrapping I can calculate confidence interval on the predicted value in a given point x. But I need a confidence interval on the error, independently of x.

  • $\begingroup$ It is common that you can not get what you want. Generally, the CI based on the regression is the function of $x$. $\endgroup$ – user158565 Jun 21 '19 at 2:51
  • $\begingroup$ Why I can get it for classification but not for regression? $\endgroup$ – aburkov Jun 21 '19 at 3:02
  • 1
    $\begingroup$ Why not just compute the appropriate quantiles of the test errors? $\endgroup$ – Demetri Pananos Jun 21 '19 at 3:50

You can try to get the distribution of the test error in the following manner:

  1. Randomly divide the dataset into training(plus cross-validation) and test samples.

  2. Estimate the model on the training (plus cross-validation) sample. Estimate the prediction error on the test sample.

  3. Repeat steps 1 and 2 a large number of times. This will give you a distribution of the prediction error. Then, you can generate the confidence interval of the prediction error from this distribution.

| cite | improve this answer | |
  • $\begingroup$ What is this "large number of times"? Also how many examples the dataset must have for this technique to work? I believe that 5 examples would not be enough. 10 either. What is a reasonable number? $\endgroup$ – aburkov Jun 27 '19 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.