42

One is a prediction of a future observation, and the other is a predicted mean response. I will give a more detailed answer to hopefully explain the difference and where it comes from, as well as how this difference manifests itself in wider intervals for prediction than for confidence. This example might illustrate the difference between confidence and ...


41

I found the following explanation helpful: Confidence intervals tell you about how well you have determined the mean. Assume that the data really are randomly sampled from a Gaussian distribution. If you do this many times, and calculate a confidence interval of the mean from each sample, you'd expect about 95 % of those intervals to include the ...


32

Take a regression model with $N$ observations and $k$ regressors: $$\mathbf{y=X\beta+u} \newcommand{\Var}{\rm Var}$$ Given a vector $\mathbf{x_0}$, the predicted value for that observation would be $$E[y \vert \mathbf{x_0}]=\hat y_0 = \mathbf{x_0} \hat \beta.$$ A consistent estimator of the variance of this prediction is $$\hat V_p=s^2 \cdot \mathbf{x_0} \...


30

@whuber has pointed you to three good answers, but perhaps I can still write something of value. Your explicit question, as I understand it, is: Given my fitted model, $\hat y_i=\hat mx_i + \hat b$ (notice I added 'hats'), and assuming my residuals are normally distributed, $\mathcal N(0, \hat\sigma^2_e)$, can I predict that an as yet unobserved response,...


26

The method laid out below is the one described in Section 6.3.3 of Davidson and Hinckley (1997), Bootstrap Methods and Their Application. Thanks to Glen_b and his comment here. Given that there were several questions on Cross Validated on this topic, I thought it was worth writing up. The linear regression model is: \begin{align} Y_i &= X_i\beta+\...


22

Ok, let's try this. I'll give two answers - the Bayesian one, which is in my opinion simple and natural, and one of the possible frequentist ones. Bayesian solution We assume a Beta prior on $p$, i,e., $p \sim Beta(\alpha,\beta)$, because the Beta-Binomial model is conjugate, which means that the posterior distribution is also a Beta distribution with ...


21

This question and excellent exchange was the impetus for creating the predictInterval function in the merTools package. bootMer is the way to go, but for some problems it is not feasible computationally to generate bootstrapped refits of the whole model (in cases where the model is large). In those cases, predictInterval is designed to use the arm::sim ...


20

It sounds like you are looking for a prediction-interval, i.e., an interval that contains a prespecified percentage of future realizations. (Look at the tag wikis for prediction-interval and confidence-interval for the difference.) Your best bet is likely to work directly with NN architectures that do not output single point predictions, but entire ...


19

Mathematics are reality-agnostic. So your negative lower prediction band can certainly be mathematically sound. I would argue, however, that this is a good indication that you are using the wrong mathematics, e.g., Ordinary Least Squares (which assumes a normal distribution of errors) with count data (where a normal distribution makes no sense). I would ...


17

Do this by making bootMer generate a set of predictions for each parametric bootstrap replicate: predFun <- function(fit) { predict(fit,newDat) } bb <- bootMer(lme1,nsim=200,FUN=predFun,seed=101) The output of bootMer is in a not-terribly-transparent "boot" object, but we can get the raw predictions out of the $t component. How much of the time ...


16

Your predict.lm code is calculating confidence intervals for the fitted values. Your hand calculation is calculating prediction intervals for new data. If you want to get the same result from predict.lm that you got from the hand calculation then change interval="confidence" to interval="prediction"


14

They live in different spaces and mean different things. A credible interval $[a,b]$ is a subset of the parameter space such that $$ P(a\leq\Theta\leq b\mid X_1=x_1,\dots,X_n=x_n) = \alpha \, , $$ and it means that, after seeing the data, you believe that with probability $\alpha$ the parameter value is inside this interval. A prediction interval $[u,v]$ ...


14

It suggests to me that you haven't used any analytic approach with an appropriate transformation of the outcome. With count data, for instance, popular linear models (Poisson Regression or Negative Binomial Regression in particular) model the log of the process as a linear function of predictors. Then, any predicted values resulting from such a model would ...


13

Your definitions appear to be correct. The book to consult about these matters is Statistical Intervals (Gerald Hahn & William Meeker), 1991. I quote: A prediction interval for a single future observation is an interval that will, with a specified degree of confidence, contain the next (or some other prespecified) randomly selected observation from a ...


12

Short answer: A prediction interval is an interval associated with a random variable yet to be observed (forecasting). A confidence interval is an interval associated with a parameter and is a frequentist concept. Check full answer here from Rob Hyndman, the creator of forecast package in R.


12

I don't know how to do prediction bands with the original loess function but there is a function loess.sd in the msir package that does just that! Almost verbatim from the msir documentation: library(msir) data(cars) # Calculates and plots a 1.96 * SD prediction band, that is, # a 95% prediction band l <- loess.sd(cars, nsigma = 1.96) plot(cars, main = "...


12

I will answer your question inside the Bayesian framework. If you specifically need a frequentist solution, you can get one by slightly modifying my answer, but I think it will underestimate the actual uncertainty: you would need a fully frequentist approach, but I don't know how to do that in this specific case. To briefly recap the Bayesian GPR (Gaussian ...


10

You need to make some assumptions about the distribution of each prediction, but since your confidence intervals are symmetric I am writing the Gaussian distribution case. For a Gaussian RV the 80 percentile corresponds to being $\approx 0.8416 \sigma$ away from the mean, so the standard deviation for each prediction can be calculated as $\sigma = (q_{(0.80)...


10

You have what is called intermittent demand, that is, a demand time series characterized by "many" zeros. (If your time series is not demand per se, most of what follows will still apply.) So a web search for "forecasting intermittent demand" would already be helpful. Teunter and Duncan (2009, JORS) give an overview of intermittent demand forecasting methods....


8

Estimation is always for unknown parameter whereas prediction is for random variable.


8

Confidence and prediction bands should be expected to typically get wider near the ends - and for the same reason that they always do so in ordinary regression; generally the parameter uncertainty leads to wider intervals near the ends than in the middle You can see this by simulation easily enough, either by simulating data from a given model, or by ...


8

There is a simple nonparametric prediction limit. Recall that a prediction limit is a procedure consisting of two independent samples $\mathcal{X}=x_1,\ldots, x_n$ and $\mathcal{Y}=y_1, \ldots, y_m$, two statistics $t$ and $s$, and a size $1-\alpha$. When the chance that $s(\mathcal{Y})$ is less than $t(\mathcal{X})$ is $\alpha$ or smaller, we say that $t$ ...


8

I'm going to start by cleaning up a little notational lack of precision in the first part of the question, then go on to the meat of it. First, if $y \in \{0,1\}$, $y$ is, more precisely than stated in the question, distributed Bernoulli$(\theta)$, not Binomial$(n,\theta)$ for some arbitrary $n$. The two distributions are of course the same when $n=1$, but ...


7

Confidence intervals take account of the estimation uncertainty. Prediction intervals add to this the fundamental uncertainty. R's predict.lm will give you the prediction interval for a linear model. From there, all you have to do is run it repeatedly on bootstrapped samples. n <- 100 n.bs <- 30 dat <- data.frame( x<-runif(n), y=x+runif(n) ) ...


7

In short, no, you don't just add the limits. Maybe if the predictions were perfectly correlated, but that's not usually the case at all. Typically (if the model assumes independence) and you want an interval for a sum of predicted values, you might then think that you can treat the predictions as independent, but they generally aren't independent even when ...


7

No, a prediction interval need not contain the mean. I think some of your confusion might be mixing prediction intervals and confidence intervals. While the goal of a prediction interval is to contain with some certainty future values of the random variable, the goal of a confidence interval is to contain the true mean of distribution. As you mentioned in ...


7

R's predict ought to be able to do a confidence interval for a GLM but definitely won't do a prediction interval -- there's an underlying statistical issue here, which I'll discuss in this answer. i) For some GLMs it doesn't make sense to even try to do a PI - consider a logistic regression with 0/1 responses, and imagine you want say a 95% PI. Anywhere ...


7

Is it? I have seen someone compute a confidence interval for the mean, and use it as if it was a prediction interval for a future observation. The trouble is, confidence intervals for the mean are much narrower than prediction intervals, and so this gave him an exaggerated and false sense of the accuracy of his forecasts. Instead of the interval ...


7

You must use the standard deviation of the prediction errors, rather than the standard deviation of the residuals. The former varies for each forecast period. For example, in an AR(1) model $$y_t=\phi y_{t-1} + \varepsilon_t \,, \quad \varepsilon_t\sim NID(0, \sigma^2) \,, \quad t=1,\dots,T\,,$$ the variance of the prediction errors can be obtained as ...


Only top voted, non community-wiki answers of a minimum length are eligible