# Sampling from Bayesian regression predictive posterior

I have the following problem: I want to obtain a predictive posterior distribution for the target logistic regression variable $y$. That is to say, given a combination of explanatory variables $X$, I want to obtain the conditional distribution $P(y|X)$ from the model.

Am I correctly supposing that I need to sample a Bayesian model via MCMC to correctly approximate the predictive posterior? Or is there any easier way? What would the best approach to such a problem be?

Also, given that I effectively have thousands of dummy explanatory variables, is it even possible to handle such a huge parameter number through sampling methods?

I would really appreciate any help here.

Am I correctly supposing that I need to sample a Bayesian model via MCMC to correctly approximate the predictive posterior? Or is there any easier way? What would the best approach to such a problem be?

Your tag says 'logistic regression', so I'm going to assume that your are trying to do Bayesian logistic OR probit regression through some data augmentation scheme. However, what I'm about to say should apply to any model where conjugacy is not an option and you are forced to turn to MCMC.

The only way to correctly (as you put it) study the posterior-predictive distribution is to take into account uncertainty in the regression coefficients & other parameters. The only clear way to do this after using MCMC to get posterior distributions on parameters is to use your MCMC samples. Any other approach would be a (potentially gross) mischaracterization of uncertainty. For example, taking the posterior means on regression coefficients & other parameters and then using your $X$ values would give you a distribution on potential $y \mid X$ values, but this distribution would not contain information about our uncertainty in the regression coefficients & other parameters. It is important that our posterior-predictive variance be inflated by our uncertainty in the parameters. This is seen in the integral $$\pi(y\mid X) = \int \! \pi(y\mid \theta, X) \pi(\theta)\,d\theta.$$ Uncertainty in $\theta$ will cause the relative tails of $\pi(\theta)$ to be valued more than $0$, and thus the relative tails of $\pi(y \mid X)$ will be valued more than $0$. Using your MCMC samples for the parameters would take into account this uncertainty and would approximate the posterior-predictive with similar Monte Carlo error.

Also, given that I effectively have thousands of dummy explanatory variables, is it even possible to handle such a huge parameter number through sampling methods?

I assume by thousands you mean under billions, in which case I would expect any modern computer to handle a sparse vector inner product efficiently & quickly. But perhaps this is not the case for you, or you just see this as unnecessarily inefficient.

One approach to dealing with this in a statistical sense is to perform variable selection on your model. Selecting the most important variables would (potentially dramatically) reduce the cost of calculating the posterior predictive at each step. Model selection in regression is conceptually straightforward in the Bayesian paradigm, but it is not necessarily quick to describe or implement. I refer you to section 9.3 of this book, but probably any book or outline of Bayesian model selection for regression would suffice.

• Thank you. The current problem is that given my dataset (I'm using STAN) and the number of variables, in order to get 1000 samples after 1000-step burn-in it takes almost two hours which isn't acceptable for the purpose at hand. You have mentioned that you have to turn to MCMC when conjugacy is not an option. Well, is conjugacy an option for logistic regression? Could anyone point me to a good resource on doing regressions via updating conjugate priors? May 19, 2014 at 9:07
• @BrunoW Unfortunately there is no conjugate prior for logistic regression. Perhaps you could post your model specification in your question to see if that is a problem? Also, if you're not attached to Bayesian methodology the algorithm used in Frequentist logistic regression would be much faster and would most likely be nearly identical if your data set is long; i.e. many rows (which it seems to be). Finally you could just subset your data (reduce the number of rows) if you aren't as concerned about being confident in your predictions.
– user44764
May 19, 2014 at 13:11
• I'm afraid I'm not able to share the code because I'm not the author. I have to use Bayesian framework because the purpose of all this is to specifically be able to sample from the predictive posterior distribution. And I'm very much concerned about the exact amount of confidence in the predictions (again, this is the goal here). Anyhow, thanks for clearing the matter up for me, I would upvote your answer but I don't have enough reputation at the moment... I'll try to remember to do it someday. May 19, 2014 at 14:44