How to visualize Bayesian goodness of fit for logistic regression

Question

For a Bayesian logistic regression problem, I have created a posterior predictive distribution. I sample from the predictive distribution and receive thousands of samples of (0,1) for each observation I have. Visualizing the goodness-of-fit is less than interesting, for example:

This plot shows the 10 000 samples + the observed datum point (way in the left one can make out a red line: yea that's the observation). The problem is is that this plot is hardly informative, and I'll have 23 of them, one for each data point.

Is there a better way to visualize the 23 data points plus there posterior samples.

---

Another attempt:

---

Another attempt based on the paper here

Answer 1

I have a feeling your not quite giving up all the goods to your situation, but given what we have in front of us lets consider the utility of a simple dot-plot to display the information.

The only real thing to not here (that aren't perhaps default behaviors) are:

• I utilized redundant encodings, shape and color, to discriminate between the observed values of no defects and defects. With such simple information, placing a dot on the graph is not necessary. Also you have a problem when the point is near the middle values, it takes more look-up to see if the observed value is either zero or one.
• I sorted the graphic according to observed proportion.

Sorting is the real kicker for dot-plots like these. Sorting by values of proportion here helps easily uncover high residual observations. Having a system where you can easily sort by values either contained in the plot or in external characteristics of the cases is the best way to get the bang for your buck.

This advice extends to continuous observations as well. You could color/shape the points according to whether the residual is negative or positive, and then size the point according to the absolute (or squared) residual. This is IMO not necessary here though because of the simplicity of the observed values.

Answer 2

The usual way to visualising the fit of a Bayesian logistic regression model with one predictor is to plot the predictive distribution together with the corresponding proportions. (Please, let me know if I understood your question)

An example using the popular Bliss' data set.

Code Below in R:

library(mcmc)

# Beetle data

ni = c(59, 60, 62, 56, 63, 59, 62, 60) # Number of individuals
no = c(6, 13, 18, 28, 52, 53, 61, 60) # Observed successes
dose = c(1.6907, 1.7242, 1.7552, 1.7842, 1.8113, 1.8369, 1.8610, 1.8839) # dose

dat = cbind(dose,ni,no)

ns = length(dat[,1])

# Log-posterior using a uniform prior on the parameters

logpost = function(par){
var = dat[,3]*log(plogis(par[1]+par[2]*dat[,1])) + (dat[,2]-dat[,3])*log(1-plogis(par[1]+par[2]*dat[,1]))

if( par[1]>-100000 ) return( sum(var) )
else return(-Inf)
}

# Metropolis-Hastings
N = 60000

samp <- metrop(logpost, scale = .35, initial = c(-60,33), nbatch = N)

samp$accept

burnin = 10000
thinning = 50

ind = seq(burnin,N,thinning)

mu1p =   samp$batch[ , 1][ind]

mu2p =   samp$batch[ , 2][ind]


# Visual tool

points = no/ni
# Predictive dose-response curve
DRL <- function(d) return(mean(plogis(mu1p+mu2p*d)))
DRLV = Vectorize(DRL)

v <- seq(1.55,2,length.out=55)
FL = DRLV(v)

plot(v,FL,type="l",xlab="dose",ylab="response")
points(dose,points,lwd=2)

Answer 3

I am responding to a request for alternative graphical techniques that show how well simulated failure events match observed failure events. The question arose in "Probabilistic Programming and Bayesian Methods for Hackers " found here. Here's my graphical approach:

Code found here.

Answer 4

I arrive here after reading "Probabilistic Programming and Bayesian Methods for Hackers". Disclaimer not sure if I'm committing a crime here, but as we have a "prediction" model with a classification problem, one way to evaluate and visualize its goodness is to apply some standard ML classification eval plots. I usually use AUC and AUCPR plots, but there's also DET curve as well. we base these plots on the posterior_probability given by the sampling procedure.

Another alternative (the one that I like the most) is the "precision-recall threshold curves" you can clearly see here how "good" your prediction probability is, given a different probability threshold. ( I think is the closest to the one that is in the book and publication)

if you also define some probability threshold you can evaluate the confusion matrix among other metrics. At least for me, it is clearer when I have the AUC score, but again, maybe I'm getting too ahead of myself using this method for a bayesian model.

code here