Estimating quantiles with linear regression?

Question

It's often stated that Gaussian Processes are probabilistic models because they not only give an estimate of $E[Y|X=x]$, but also of the variability around this mean function. However, isn't this true of all (most?) statistical models? Consider simple linear regression with a sample of size $n$:

$y = \beta_0 + \beta_1x+\epsilon$

Suppose that, for a certain $x$, I want to estimate $y_{q} \ s.t. P\left(y(x)<y_{q}\right)=q$. This is given by the upper bound of the prediction interval:

$y_q(x)=\hat{y}+t_{q,n-2}s_y\sqrt{1+\frac{1}{n}+\frac{x-\bar{x}}{(n-1)s^2_x}}=\left(\hat{\beta_0}+\hat{\beta_1}x\right)+t_{q,n-2}s_y\sqrt{1+\frac{1}{n}+\frac{x-\bar{x}}{(n-1)s^2_x}}$

where $t_{q,n-2}$ is the $q$-quantile of the $t$-distribution with $n-2$ degrees of freedom, $s_y$ is the standard deviation of the residuals, etc.

1. is this conceptually correct?
2. as long as we can define a prediction interval for a given statistical model, we can estimate (conditional) quantiles this way. Thus, it seems to me that most statistical models allow to estimate not only the conditional mean, but also the conditional response distribution - it's not a prerogative of Gaussian Processes. Correct?

Answer 1

Your logic is more suited to the Bayesian rather than classical statistical setting. Note that in your interval the randomness applies to the lower and upper bounds of the interval, as these are considered to vary over (infinitely many) random samples. Thus your probability statement about the interval should really be read as $\Pr( [L(y), U(y)] \mbox{ contains } y^* | \mu, \sigma)$. In words, the chance that any random sample's prediction interval contains a new value (generated from the same distribution as the data), which depends on unknown but fixed parameters.

So you don't have an interval about $y^*_q$.

I suggest you look at recent clarification of correct interpretation of classical intervals (eg p-values) in Wasserstein and Lazar (2016), on behalf of the American Statistical Association. Some good references about long running debate.

Bayesians are often pointing out the way in which Frequentist (classical statistical) intervals are misinterpreted. Many people provide an interpretation more akin to a Bayesian posterior probability, which for you would require the posterior predictive distribution $\Pr(y^*| y) = \int_\theta p(y^*|\theta) p(\theta|y) d\theta$, where $\theta=(\mu,\sigma)$. You could then take any summary statistic, such as quantiles from this distribution. An excellent mathematical intro is provided by Gelman, Carlin, Stern and Rubin's seminal book "Bayesian Data Analysis".

Alternatively investigate topics of "tolerance intervals" (classical stats needs extra contortions to get probability statements about confidence bounds) or "quartile regression" to model quantiles.