I have a model which gives success probability estimates for a sequence of non-identically distributed Bernoulli trials. When the model is not constrained to be well calibrated with respect to observed frequencies (e.g., possibly models not fitted with MLE), model probability calibration can be necessary, and the procedure starts by looking at places where the model is miscalibrated.
The way I have been doing this is by binning the probabilities and plot expected frequency vs observed frequency in each bin. This post describes such a procedure well. Other places recommend bins with equal probability width, say 0 - 0.1, 0.1 - 0.2, etc. I am wondering whether there are statistical tests instead that can tell you the degree to which the model is miscalibrated by, perhaps a hypothesis test that can generate a p-value against the null hypothesis "the model is well calibrated".
For example, would it be possible to adapt the chi-squared test to this purpose: Say we have 10 bins of equal 0.1 width, thus the mean probability in each bin is 0.05, 0.15, 0.25, etc. In each bin we have the number of observations falling in the bin, which we can use to normalise the bin probabilities (so that they sum to 1). Then we can apply the chi-squared test with number of successes in each bin vs the normalised bin probabilities.
