3
$\begingroup$

I want to generate predictions from a fitted logistic regression model that reflect the uncertainty of the model (within a classic frequentist framework). To clarify, my objective is not to characterize the predictive uncertainty as such, but to get a set of reasonable predictions which are fair to the uncertainty about the coefficients of the predictors included in the model .

My approach is as follows: for each group of interest, defined by a combination of values of the predictors; draw values for each regression coefficient from a normal distribution with a mean - the point estimate of the respective regression coefficient, and a standard deviation - the respective standard error of the regression coefficient; plug the resulting values in the inverse logit formula to get the prediction; and repeat. Would that be a valid approach? Would it work for mixed-effects (multilevel) models as well where the predictors are modeled as random effects?

$\endgroup$
2
$\begingroup$

In the frequentist framework it suffices to compute the standard error of $X\hat{\beta}$, get a confidence interval from this for $X\beta$, and use the logistic transformation of those two confidence limits to get a confidence interval for $P$ for an individual with characteristics $X$. I believe an adaptation of this approach will work for mixed effects models but have not done that myself.

$\endgroup$
  • $\begingroup$ but then I would have to sample the actual prediction sets from the confidence interval. so would my approach, which appears more direct to code with many combinations of predictors, still work? $\endgroup$ – Dimiter May 7 '14 at 12:17
  • $\begingroup$ No, there is no reason to sample anything to solve the problem that you started the discussion with. Just provide the 0.95 confidence interval for the true risk. $\endgroup$ – Frank Harrell May 7 '14 at 16:28
  • 1
    $\begingroup$ If you still want a sampling approach you can use the "poor man's Bayesian approach" of bootstrapping whereby you develop (from scratch) several models from samples with replacement from the original dataset and show the distributions of predicted values. The 0.025 and 0.975 quantiles of predictions for one subject would provide a 0.95 confidence interval for the unknown true probability of an event for that subject, which is the long way to obtain a confidence interval, but sometimes more accurate than Gaussian theory-based Wald intervals. $\endgroup$ – Frank Harrell Oct 21 '14 at 10:20
1
$\begingroup$

Rather than generating each regression coefficient separately from individual normal distributions, shouldn't you use a multivariate normal distribution, since you should also have covariances from you original fitted model? This is important because it takes into account the probability of different combinations of coefficients occuring together.

I know this doesn't really answer the question you were asking, but I think it is an important point.

$\endgroup$
  • 1
    $\begingroup$ Good point! Welcome to CV! The parameters of the model are not independent. @Dimiter in frequentionist approach the uncertainty is about the sampling, no the parameters. If you are interested in obtaining a distribution of parameters, and threat the parameters as draws from some distribution then you probably should consider Bayesian approach. $\endgroup$ – Tim Dec 8 '14 at 13:57
  • $\begingroup$ yeah, that's what I was afraid of. $\endgroup$ – Dimiter Dec 9 '14 at 14:03
1
$\begingroup$

In frequentionist approach the estimate from the modes is the best possible estimate given the data you have. Of course, your analysis needs to meet all the assumptions of the procedure you used and you have to make your math correctly. Your estimate however is dependent on your data, so there is some possible uncertainty because of the sampling. If you sampled all the population of the possible outcomes of your "experiment" you would get an unbiased estimate. In practice it is impossible, so you have to deal with the uncertainty in some way. The most common approach is to calculate Confidence Intervals around your estimates. You can obtain them based doing some calculations (see answer by Frank Harrell) based on assumptions of your model (i.e. analytically).

On another hand, there are some situations where it is hard to obtain Confidence Intervals analytically. One approach in here is to use bootstrap, that is: sample multiple samples with replacement from your dataset, estimate your model on those samples and in the end you obtain a distribution of possible estimates, that can be used to estimate the uncertainty of your result. So the main idea is here to sample from the data in the similar fashion how you sampled from the population, what helps you to get some insight on uncertainty due to the sampling. In practice, this is a little bit more complicated, but the main idea is the same. It gets more complicated with hierarchical data (e.g. mixed effects models), but that is another story.

There is however the third approach: Bayesian. From your question I understand that what you believe is that there is some uncertainty of your estimate - Bayesians feel the same way! In Bayesian viewpoint, everything is uncertain and you incorporate this uncertainty in your statistical model. You do this by picking some (subjective) a priori distribution of estimates you find valid, then Bayes theorem lets you confront those assumptions with your data and obtain a distribution of estimates that are most likely. So you assume that there could be some variability in your data, but also in your estimation and you obtain an estimate that incorporates this uncertainty. I know that you asked for classical solution, but this one is the only valid if you really want to attain for uncertainty of estimates of the parameters.

The approach you described is not valid because to sample your coefficients you would need standard errors for them and if you have them there is no need for sampling because you can calculate the confidence intervals directly using the standard errors. Also, as Amy Spencer noticed, coefficients are correlated so drawing the independently would give you biased estimates. If you really want to resample, there is a better alternative: bootstrapping residuals (check e.g. Davison and Hinkley, 1997).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.