# Metrics for assessing the quality of prior distributions

Clarification: My purpose is to compare different methods for selecting/creating priors (or perhaps I should refer to them as predictive distributions for a quantity of interest/parameter). I do not want to select a prior for the particular set of data for which I have the likelihood, but rather I want to figure out in some sense what way of setting up a prior (or predictive distribution) gave the prior that best predicted the data / the likelihood. I'd probably want to ideally do this several time and then look in a frequentist sense at which approach seems to work best when I repeatedly need to create predictive distributions in similar settings.

Original main question: Let's say I have a number of prior distributions (e.g. from different approaches to getting prior distributions or from different experts) for a quantity of interest and that I have now done an experiment to determine the value of the quantity of interest. I want to figure out which priors were the "best". I'm struggeling with how one should actually define "best" in this context.

Let's illustrate this with some hypothetical priors and a likelihood from the experiment. I.e. I don't get a single correct value, but instead e.g. have an estimate and a standard error. Let's use this plot as an example: It seems relatively clear that we would

• favor priors 4 and 5 over 1 and 2, because they are more certain and not proven wrong by the data, and
• favor priors 1, 2, 4 and 5 over prior 3, because prior 3 has the bulk of the probability mass in what seems to be a region not very compatible with the data.

A metric like the Hellinger distance (between the priors and the normalized likelihood) would capture those judgments of mine. However, it seems like that's not really such a good metric. E.g. it would strongly favor prior 5 over prior 4, which is, not at all, a clear case. This also becomes more obvious when we have less data and e.g. the likelihood is wider (and e.g. looks like prior 2). It seems wrong to me that prior 4 or 5 would become "worse", because we have less data (but it actually does not contradict the priors, at all). Sure, in the case where the likelihood looks more like prior 2, prior 2 should be considered better than under the likelihood shown above in black, but we could not really say that the data contradicts prior 4 or 5, could we?

I suspect someone must have done research on this topic and/or published about it, but somehow I failed to find the right search terms to find how people typically approach this. Any answers on sensible ways of approaching this problem and/or pointers literature would be appreciated.

• I think that using the data to select a prior is not acceptable but Michael Evans has written a book on the opposite perspective. – Xi'an Jun 1 at 7:17
• @Xi'an Sorry, I should have been clearer, I do not intend to select a prior for the particular set of data, I'm trying to compare different methods for selecting/creating priors. I.e. it's not about the analysis of this data, but rather for deciding that one way of creating priors is usually better (for some definitions of usually and better - usually probably = on average for some metric, but the better = what metric is what I'm struggeling with). – Björn Jun 1 at 7:26
• "Best" in what sense? Do you mean the most "uninformative"? Why not using maximum likelihood? – Tim Jun 1 at 7:53
• @Tim, one example would be expert elicitation methods. For example, we might want to try several different approaches: e.g. an elicitation workshop using HENVINET vs. SHELF vs. a modified Delphi approach vs. ELI vs. Cooke vs. maybe some automated machine learning approach. We might wonder which of these is best in some sense (i.e. most closely predicts the true value, which we can only measure with noise in an experiment of practical size). Ideally, we'd want the most informative prior that still gives high probability density at (and near) the true value. – Björn Jun 1 at 8:00
• But you choose a prior before collecting the data. You can base it on past experiments, ask experts, etc, but such prior cannot be judged in terms of the unseen data. If you want a data-optimal prior, you could use something like empirical Bayes, but as noticed by @Xi'an it is a disputable practice. You could use "uninformative" ones. But with informative priors it sounds like wanting to have a cake and eat a cake. – Tim Jun 1 at 9:45