I have a binary classification problem (i.e. is observation positive or negative) and I'm interested in what information I can obtain about observations in my test set. I don't care about the model parameters or observations in my training set. I have plenty of data and would just use an uninformative prior as the task is fairly simple.

With a frequentist logistic regression I can only get a probability estimate of whether the observation is positive/negative, but from what I've heard Bayesian methods can give more information such as an uncertainty estimate. With regards to this I have the following questions:

  1. How does the Bayesian method obtain this extra information? Can you refer to any material on this?

  2. What extra information can be obtained at prediction time? Can one get more information than an uncertainty estimate?

  • $\begingroup$ Your question is contradictory: on one hand you say that you don't care about the parameters, on another, you are interested in learning "extra information" about your model -- so what exactly do you want..? $\endgroup$ – Tim Aug 6 '17 at 20:07
  • $\begingroup$ It seems like a reasonable (and non-contradictory) need to get e.g. uncertainty estimates on a black box model's predictions? Can you elaborate on why it's contradictory? $\endgroup$ – pir Aug 6 '17 at 20:33
  • $\begingroup$ What more then "uncertainty estimate" would you need? $\endgroup$ – Tim Aug 6 '17 at 20:50
  • $\begingroup$ I didn't know the concept of posterior predictive distribution as I've seen peeople collapse it into a single number for a given prediction that they called an uncertainty estimate. The posterior predictive distribution is what I need. $\endgroup$ – pir Aug 6 '17 at 20:57

You can start with the Bayesian logit model - intuitive explanation? thread, or this one about linear regression. The basic difference is that with Bayesian model you get a full probabilistic model, so what the model returns is the posterior distributions of your parameters and posterior predictive distribution of the predictions. Additional advantage is that you can include some prior information in your model, but if you plan to use "uninformative" prior, you do not seem to be interested in such advantage.

So the basic difference for your purpose is that in frequentist case you would obtain your predictions and you could estimate confidence intervals, or prediction intervals. In Bayesian case, given your model you could estimate and simulate the whole posterior predictive distribution, so instead of interval estimates you could check many more properties of the distribution. Notice however that in frequentist setting you could also simulate the potential predictions from your model to approximate the posterior predictive distribution, or use bootstrap for similar purpose.

  • $\begingroup$ Do you have any advice for interpreting the postior predictive distribution when you have multiple input features in your model? $\endgroup$ – pir Aug 6 '17 at 21:29
  • $\begingroup$ @pir what exactly would you like to interpret? This is just the distribution of possible values given your model. $\endgroup$ – Tim Aug 7 '17 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.