What non-Bayesian methods are there for predictive inference?

Question

In Bayesian inference a predictive distribution for future data is derived by integrating out unknown parameters; integrating over the posterior distribution of those parameters gives a posterior predictive distribution—a distribution for future data conditional on those already observed. What non-Bayesian methods for predictive inference are there that take into account uncertainty in parameter estimates (i.e. that don't merely plug maximum-likelihood estimates or whatever back into a density function) ?

Everyone knows how to calculate prediction intervals after a linear regression, but what are the principles behind the calculation & how can they be applied in other situations (e.g. calculating an exact prediction interval for a new exponential variate after estimating the rate parameter from data)?

Answer 1

Non-Bayesian predictive inference (apart from the SLR case) is a relatively recent field. Under the heading of "non-Bayesian" we can subdivide the approaches into those that are "classical" frequentist vs those that are "likelihood" based.

Classical Frequentist Prediction

As you know, the "gold standard" in frequentism is to achieve the nominal coverage under repeated sampling. For example, we want a 95% Confidence Region to contain the true parameter(s) in 95% of samples from the same underlying population. Or, we expect to commit Type I and II errors in a hypothesis test on average equal to $\alpha$ and $\beta$. Finally, and most germane to this question, we expect our 95% Prediction Interval to contain the next sample point 95% of the time.

Now, I've generally had issues with how classical PI's are presented and taught in most stats courses, because the overwhelming tendency is to interpret these as Bayesian posterior predictive intervals, which they are decidedly not. Most fundamentally, they are talking about different probabilities! Bayesian's make no claim on the repeated sampling performance of their quantities (otherwise, they'd be frequentists). Second, a Bayesian PI is actually accomplishing something more similar in spirit to a Classical Tolerance Interval than to a Classical Prediction Interval.

For reference: Tolerance Intervals need to be specified by two probabilities: The confidence and the coverage. The confidence tells us how often it is correct in repeated samples. The coverage tells us the minimum probability measure of the interval under the true distribution (as opposed to the PI, which gives the expected probability measure...again under repeated sampling). This is basically what the Bayesian PI is trying to do as well, but without any repeated-sampling claims.

So, the basic logic of the Stats 101 Simple Linear Regression is to derive the repeated sampling properties of the PI under the assumption of normality. Its the frequentist+Gaussian approach that is typically thought of as "classical" and taught in intro stats classes. This is based on the simplicity of the resulting calculations (see Wikipedia for a nice overview).

Non-gaussian probability distributions are generally problematic because they can lack pivotal quantities that can be neatly inverted to get an interval. Therefore, there is no "exact" method for these distributions, often because the interval's properties depend on the true underlying parameters.

Acknowledging this inability, another class of prediction arose (and of inference and estimation) with the likelihood approach.

Likelihood-based Inference

Likelihood-based approaches, like many modern statistical concepts, can be traced back to Ronald Fisher. The basic idea of this school is that, except for special cases, our statistical inferences are on logically weaker ground than when we are dealing with inferences from a normal distribution (whose parameter estimates are orthogonal), where we can make exact probability statements. In this view of inference, one should really avoid statements about probability except in the exact case, otherwise, one should make statements about the likelihood and acknowledge that one does not know the exact probability of error (in a frequentist sense).

Therefore, we can see likelihood as akin to Bayesian probability, but without the integrability requirements or the possible confusion with frequentist probability. Its interpretation is entirely subjective...although a likelihood ratio of 0.15 is often recommended for single parameter inference.

However, one does not often see papers that explicitly give "likelihood intervals". Why? It appears that this is largely a matter of sociology, as we have all grown accustomed to probability-based confidence statements. Instead, what you often see is an author referring to an "approximate" or "asymptotic" confidence interval of such and such. These intervals are largely derived from likelihood methods, where we are relying on the asymptotic Chi-squared distribution of the likelihood ratio in much the same way we rely on the asymptotic normality of the sample mean.

With this "fix" we can now construct "approximate" 95% Confidence Regions with almost as much logical consistency as the Bayesians.

From CI to PI in the Likelihood Framework

The success and ease of the above likelihood approach led to ideas about how to extend it to prediction. A very nice survey article on this is given here (I will not reproduce its excellent coverage). It can be traced back to David Hinkley in the late 1970's (see JSTOR), who coined the term. He applied it to the perennial "Pearson's Binomial Prediction Problem". I'll summarize the basic logic.

The fundamental insight is that if we include an unobserved data point, say $y$, in our sample, and then perform traditional likelihood inference on $y$ instead of a fixed parameter, then what we get is not just a likelihood function, but a distribution (unnormalized), since the "parameter" $y$ is actually random and therefore can be logically assigned a frequentist probability. The mechanics of this for this particular problem are reviewed in the links I provided.

The basic rules for getting rid of "nuisance" parameters to get a predictive likelihood are as follows:

1. If a parameter is fixed (e.g., $\mu, \sigma$), then profile it out of the likelihood.
2. If a parameter is random (e.g., other unobserved data or "random effects"), then you integrate them out (just like in Bayesian approach).

The distinction between a fixed and random parameter is unique to likelihood inference, but has connections to mixed effects models, where it seems that the Bayesian, frequentist, and likelihood frameworks collide.

Hopefully this answered your question about the broad area of "non-Bayesian" prediction (and inference for that matter). Since hyperlinks can change, I'll also make a plug for the book "In All Likelihood: Statistical Modeling and Inference using Likelihood" which discusses the modern likelihood framework at depth, including a fair amount of the epistemological issues of likelihood vs Bayesian vs frequentist inference and prediction.

---

References

1. Prediction Intervals: Non-parametric methods. Wikipedia. Accessed 9/13/2015.
2. Bjornstad, Jan F. Predictive Likelihood: A Review. Statist. Sci. 5 (1990), no. 2, 242--254. doi:10.1214/ss/1177012175. http://projecteuclid.org/euclid.ss/1177012175.
3. David Hinkley. Predictive Likelihood. The Annals of Statistics Vol. 7, No. 4 (Jul., 1979) , pp. 718-728 Published by: Institute of Mathematical Statistics Stable URL: http://www.jstor.org/stable/2958920
4. Yudi Pawitan. In All Likelihood: Statistical Modeling and Inference Using Likelihood. Oxford University Press; 1 edition (August 30, 2001). ISBN-10: 0198507658, ISBN-13: 978-0198507659. Especially Chapters 5.5-5.9, 10, and 16.

Answer 2

I'll address my answer specifically to the question, "What non-Bayesian methods for predictive inference are there that take into account uncertainty in parameter estimates?" I will organize my answer around expanding the meaning of uncertainty.

We hope statistical analyses provide support for various kinds of claims, including predictions. But we remain uncertain about our claims, and this uncertainty arises from many sources. Frequentist statistics is characteristically organized around addressing only that part of our uncertainty arising specifically from sampling. Sampling may well have been the main source of uncertainty in the agricultural field experiments that historically provided much of the stimulus to the development of frequentist statistics. But in many of the most important current applications, this is not the case. We now worry about all kinds of other uncertainties like model misspecification and various forms of bias---of which there are apparently hundreds (!) of types[1].

Sander Greenland has a wonderful discussion paper [2] that points out how important it can be to take account for these other sources of uncertainty, and prescribes multiple-bias analysis as the means to accomplish this. He develops the theory entirely in Bayesian terms, which is natural. If one wishes to carry forward a formal, coherent treatment of one's uncertainty about model parameters, one is led naturally to posit (subjective) probability distributions over parameters; at this point you are either lost to the Bayesian Devil or have entered the Bayesian Kingdom of Heaven (depending on your religion).

To your question, @Scortchi, about whether this can be done with "non-Bayesian methods," a non-Bayesian workaround is demonstrated in [3]. But to anyone who knows enough about Bayesianism to write your question, the treatment there will look rather like an attempt to implement Bayesian calculations 'on the sly' so to speak. Indeed, as the authors acknowledge (see p. 4), the closer you get to the more advanced methods toward the end of the book, the more the methods look like precisely the integration you describe in your question. They suggest that where they depart from Bayesianism ultimately is only in not positing explicit priors on their parameters before estimating them.

To tie this in explicitly to prediction, one need only appreciate the 'prediction' as a function of the estimated parameters. In [2], Greenland uses the notation $\theta(\alpha)$, where $\alpha$ is the vector of model parameters, and $\theta$ is the function of those parameters which is to be estimated. (In terms of Greenland's example application, a meaningful prediction could be the impact in terms of reduced pediatric leukemia of a policy of relocating power lines.)

1. Chavalarias, David, and John P A Ioannidis. “Science Mapping Analysis Characterizes 235 Biases in Biomedical Research.” Journal of Clinical Epidemiology 63, no. 11 (November 2010): 1205–15. doi:10.1016/j.jclinepi.2009.12.011.

2. Greenland, Sander. “Multiple-Bias Modelling for Analysis of Observational Data (with Discussion).” Journal of the Royal Statistical Society: Series A (Statistics in Society) 168, no. 2 (March 2005): 267–306. doi:10.1111/j.1467-985X.2004.00349.x.

3. Lash, Timothy L., Matthew P. Fox, and Aliza K. Fink. Applying Quantitative Bias Analysis to Epidemiologic Data. Statistics for Biology and Health. New York, NY: Springer New York, 2009. http://link.springer.com/10.1007/978-0-387-87959-8.