Comparing posterior predicted probabilities with "known" to be true probabilities

Question

Following this description I have implemented a Bayesian logistic regression (BLR) model on some data.

Lets say I have this kind of data:

    Sex  Age Success prob
1   Male  18       1  0.2
2   Male  31       0  0.5
3 Female  27       1  0.7

And I fitted my BLR model without using prob. If we assume that these "prob" probabilities is the true probability for success is there a good, scalable, way to compare my model probabilities with these?

As far as I know, graphical methods where we draw samples from the posterior and look how far away they are from prob only works for one observation at a time. I'm looking for something that scales better, preferably something that can be automated completely or to some extend.

Here are a few ideas I have come up with:

1. Find the MAP and compare it with prob using simple linear regression

2. Turn prob into a Bayesian model and compare this model to my BLR model using Bayes factor

I'm not sure how to do this. I'm thinking that would be a single parameter Bayesian logistic model with a non informative (Jeffreys?) prior. Likelihood function would be Bernoulli($y_i; p_i$) where $y_i$ is the success outcome and $p_i$ prob.

Method 1) feels very underwhelming for some reason.

So, I have two questions:

1. What do you think about method 2, will it work as intended? Or will there be too much information loss?

2. Is there any other way to do this? Keywords to search for?

Thanks in advance.

Answer 1

I don’t exactly have an answer for you. I have done things like you are trying to do as a demonstration of the properties of statistics, but not for any type of validation. It makes me wonder what you are doing and why.

Let me begin with an observation as well. You are proposing using an uninformative prior distribution for this project. The first difficulty that may be created is that your posterior may not sum to unity. It may not end up being a probability distribution. That is only assured with proper priors. Improper priors are not guaranteed to result in proper probabilities. The second is that I am not sure what you are going to gain by doing that.

Null hypothesis methods provide a guarantee to the performance of their estimators and do not include prior information. Bayesian methods have no performance guarantees.

If you were willing to use the MAP estimator, why not the MLE? At least with the MLE, you will get guarantees on your inference, and you do not have to worry about the model exploding due to a failure to integrate to unity.

With that said, if your posterior does integrate to unity or you use a proper prior, then your estimators cannot be first-order stochastically dominated by another estimator. That is to say, any other estimator will be as risky or riskier than the Bayesian one. On average, you will be as close or closer with a Bayesian estimator.

With all of that said, if you are creating a random set of data to check performance, then you should look at two things. The first one is how do the estimators perform when the sample is reasonably representative. The second one is how do the estimators perform when the sample statistics are far away from the population parameters?

Unbiased estimators minimize the maximum amount of risk you will be exposed to should you get an unrepresentative sample, while Bayesian estimators minimize your average loss.

There is any number of potential measures against a known parameter. You could find the distribution of the MAP estimator or the posterior mean around the true value. You could find the posterior variance around the parameter set. Each of those scales well.

I don’t think either 1 or 2 will do what you want.

I would take the points around the parameters from my MCMC set and then calculate what I needed, depending on my goal. As long as your set is large enough and converges, it should do what you want.

You should strongly consider using proper a proper prior distribution with real information in it, or consider using a non-Bayesian method.