Following this description I have implemented a Bayesian logistic regression (BLR) model on some data.
Lets say I have this kind of data:
Sex Age Success prob 1 Male 18 1 0.2 2 Male 31 0 0.5 3 Female 27 1 0.7
And I fitted my BLR model without using prob. If we assume that these "prob" probabilities is the true probability for success is there a good, scalable, way to compare my model probabilities with these?
As far as I know, graphical methods where we draw samples from the posterior and look how far away they are from prob only works for one observation at a time. I'm looking for something that scales better, preferably something that can be automated completely or to some extend.
Here are a few ideas I have come up with:
Find the MAP and compare it with prob using simple linear regression
Turn prob into a Bayesian model and compare this model to my BLR model using Bayes factor
I'm not sure how to do this. I'm thinking that would be a single parameter Bayesian logistic model with a non informative (Jeffreys?) prior. Likelihood function would be Bernoulli($y_i; p_i$) where $y_i$ is the success outcome and $p_i$ prob.
Method 1) feels very underwhelming for some reason.
So, I have two questions:
What do you think about method 2, will it work as intended? Or will there be too much information loss?
Is there any other way to do this? Keywords to search for?
Thanks in advance.