How to compute a confidence interval on the regression error? I have a regression model (not necessarily linear regression) and a test set. I would like to be able to say: "If you use my model, then with probability 95%, the prediction error will be in the interval [a,b]" where the prediction error is the difference between predicted y and the real y as given by the test data.
Is there any method to estimate the values a and b from the test set?
I know that using bootstrapping I can calculate confidence interval on the predicted value in a given point x. But I need a confidence interval on the error, independently of x.
 A: When you form a prediction interval in regression, this is already a confidence interval for the true response value --- i.e., you use $\hat{y}_*$ and the rest of the regression information to get a confidence interval for the unknown value $y_*$ (for some observation $*$ with known explanatory variables).  The usual form for the prediction interval in linear regression with $k$ explanatory variables and an intercept term is:
$$\text{PI}_{Y_*}(1-\alpha) = \bigg[ \hat{y}_* \pm t_{n-k-1,\alpha/2} \sqrt{MSE + \hat{\text{se}}^2(\hat{Y}_*)} \bigg].$$
So, if you want a confidence interval for the "prediction error" $r_* \equiv \hat{y}_* - y_*$ (which is usually called the "residual"$^\dagger$) you just take the prediction interval and subtract the predicted response:
$$\text{CI}_{R_*}(1-\alpha) = \bigg[ 0 \pm t_{n-k-1,\alpha/2} \sqrt{MSE + \hat{\text{se}}^2(\hat{Y}_*)} \bigg].$$

$^\dagger$ Note that there is a distinction in regression analysis between the "error" and the "residual".  The substance of your question appears to be seeking an interval estimator for the unobserved residual, but there are other parts of the question that refer to this erroneously as the "error".
A: You can try to get the distribution of the test error in the following manner:


*

*Randomly divide the dataset into training(plus cross-validation) and test samples. 

*Estimate the model on the training (plus cross-validation) sample. Estimate the prediction error on the test sample.

*Repeat steps 1 and 2 a large number of times. This will give you a distribution of the prediction error. Then, you can generate the confidence interval of the prediction error from this distribution. 
