5
$\begingroup$

If you fit a quantile regression for the 5th and 95th percentile this is often described as an estimate of a 90% prediction interval. This is the most prevalent it seems in the machine learning domain where random forests has been adapted to predict the quantiles of each leaf node or GBM with a quantile loss function.

Is this best characterized as a confidence or prediction interval and why?

$\endgroup$
5
+50
$\begingroup$

Definitely a prediction interval, see for example here.

Quantile regression for the $5^\textrm{th}$ and $95^\textrm{th}$ quantiles attempts to find bounds $y_0({\bf x})$ and $y_1({\bf x})$, on the response variable $y$ given predictor variables ${\bf x}$, such that $$ \mathbb{P}\left(Y\le y_0({\bf X})\right)=0.05 \\ \mathbb{P}\left(Y\le y_1({\bf X})\right)=0.95 $$ so $$ \mathbb{P}\left(\,y_0({\bf X})\le Y\le y_1({\bf X})\,\right)\ =\ 0.90 $$ which is by definition a $90\%$ prediction interval.

A $90\%$ prediction interval should contain (as-yet-unseen) new data $90\%$ of the time. In contrast, a $90\%$ confidence interval for some parameter (e.g. the mean) should contain the true mean unless we were unlucky to the tune of 1-in-10 in the data used to construct the interval.

$\endgroup$
-1
$\begingroup$

After posting this I realized it is most accurately called a confidence interval, regardless of the terminology used by sci-kit learn.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/

"However, it must be kept in mind that the resulting confidence intervals are a model approximation rather than true statistics".

$\endgroup$
  • 5
    $\begingroup$ Most "accurately" called? No. Most commonly? Probably. Take a look at the documentation for the ‘randomUniformForest R package, which has a "confInt" type for the predict method which yields, according to the manual pages, prediction intervals. So there appears to be widespread confusion between the two concepts (see also the reference I gave in my main answer). I'm going to stick to my guns -- the correct terminology is prediction interval for data and confidence interval for a parameter -- but it looks like very few people will care if you get it wrong. $\endgroup$ – Creosote Sep 9 '15 at 20:12
  • 1
    $\begingroup$ Sorry but that is not correct. Confidence intervals are intervals around expected values such as E(y|X) while prediction intervals are intervals around y|X. See this stats.stackexchange.com/questions/16493/… $\endgroup$ – B_Miner Sep 9 '15 at 23:51
  • 1
    $\begingroup$ Yes, CI's are for E(Y|x), PI's for Y|x, but in the Meinshausen paper it is the latter case that we're looking at. Quote: "Prediction Intervals How reliable is a prediction for a new instance?". The new instance is Y, given explanatory variables x. The author goes on: "a new observation of Y, for X=x, is with high probability in the interval I(x)". This is a statement about Y itself, not the mean of all possible Y, given x. If you are asking in your original question if Meinshausen is using the wrong terminology then I remain quite sure that he is not. $\endgroup$ – Creosote Sep 10 '15 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.