# Assessing model prediction performance using tolerance interval coverage

I am trying to assess a model's prediction performance, and one metric I look at is the percentage of new observations that fall within the 95% tolerance interval, and see whether or not it is actually close to 95%. If the percentage of new observations that fall in the tolerance interval is much lower than 95%, could I conclude based on this metric, the model has poor prediction?

Is the method I described above widely used? If you know of any references, please list a few.

Please note: I look at other metrics as well, but I want to verify whether or not I can draw such a conclusion using this metric. I am not looking for other suggestions for assessing model prediction performance.

Are you talking about a frequentist prediction interval or a Bayesian one? The frequentist prediction interval absolutely doesn't have to contain 95% of the new observations. The frequentist prediction procedure is such that, if you collect a vast amount of samples of size $N$ and for each of these samples you form the corresponding prediction interval, then about 95% of these intervals will contain one observation not included in the $N$ ones used to build the prediction interval.

In more formal terms, if $(X_1,\dots,X_N)=\mathbf{X}$ is a random vector with independent and identically distributed components, and if $X^*$ is another random variable independent of the components of $\mathbf{X}$ but having their same distribution, then a (frequentist) 95%-confidence prediction interval is a random interval $I(\mathbf{X})=[a(\mathbf{X}),b(\mathbf{X})]$ such that

$$P[X^*\in I(\mathbf{X})]=.95$$

This means that if I collect a sample $(x_1,\dots,x_N)=\mathbf{x}$, i.e., a realization of $\mathbf{X}$, and compute the corresponding prediction interval $I(\mathbf{x})$, then another observation $x^*$ may or may not belong to $I(\mathbf{x})$. However, if I were to repeat this procedure multiple times (collect sample $\mathbf{x}$; compute $I(\mathbf{x})$; collect $x^*$), then on average $I(\mathbf{x})$ would contain $x^*$ 95% of times. Note that the confidence level for a prediction procedure refers to the percentage of prediction intervals which contain an extra observation not used to build the intervals. It does not refer to the percentage of new observations that we collect! If we build an interval and collect $x^*=x_1^*$, we're 95%-confident that $I(\mathbf{x})$ will contain $x_1^*$. Now, suppose we keep collecting more new observations $x^*_2,\dots,x^*_m,\dots$: we cannot say that we are 95% confident that 95%, or 50%, or 99% or whatever percentage of the new observations will fall in $I(\mathbf{x})$. We just don't know.

If I understand your question correctly, you're looking for something different: you need an interval which, 95% of times, would contain 95% of the population. In other words, you need a procedure which, for each sample $\mathbf{x}$, builds an interval $T(\mathbf{x})$ such that, for 95% of all possible samples, has the following property:

• as you keep collecting more and more observations, not used to build $T(\mathbf{x})$, the percentage of observations which falls in $T(\mathbf{x})$ tends to 95%, as the number of observations goes to $\infty$.

Of course, for a given sample $\mathbf{x}$ and corresponding interval $T(\mathbf{x})$, you cannot be sure that $T(\mathbf{x})$ will contain 95% of the population. However, you know that, if you repeat the procedure more and more times, on average 95% of the intervals will be "good". Such intervals are called tolerance intervals: the R package tolerance helps you build them.

As a final note, I personally have never heard of tolerance intervals being used to assess the predictive capability of a statistical model, but of course that doesn't mean that it has never been done, or that it's a bad idea.

PS to keep things simple, when talking about tolerance intervals I considered the confidence level of the procedure and the percentage of the population covered to be numerically equal. However, that's not needed: you can define a 95%-confidence tolerance interval for the central 50% of the population, or a 50%-confidence tolerance interval for the central 95% of the population (though I don't think anyone would ever use a 50% confidence procedure!).

• Thank you very much for your detailed answer and for clarifying what I am looking for. I would like to receive some further feedback on how widely used this method is. I will upvote your answer when I have enough rep... – Laura Lels Oct 22 '16 at 14:21
• no need for upvoting, you can just accept the answer you prefer. Concerning feedback, I cannot offer any because, as I said, I never heard of tolerance intervals being used to assess the predictive performance of a method. – DeltaIV Oct 22 '16 at 15:37

There is no relation between a 'correct' coverage probability and predictive performance. Take e.g. as prediction interval the range from the 2.5% to the 97.5% quantile of the responses in the training set. Its coverage probability will probably not be too far away from 95% in an independent test set but the predictive power of this empty model will be poor.