Clarification: My purpose is to compare different methods for selecting/creating priors (or perhaps I should refer to them as predictive distributions for a quantity of interest/parameter). I do not want to select a prior for the particular set of data for which I have the likelihood, but rather I want to figure out in some sense what way of setting up a prior (or predictive distribution) gave the prior that best predicted the data / the likelihood. I'd probably want to ideally do this several time and then look in a frequentist sense at which approach seems to work best when I repeatedly need to create predictive distributions in similar settings.
Original main question: Let's say I have a number of prior distributions (e.g. from different approaches to getting prior distributions or from different experts) for a quantity of interest and that I have now done an experiment to determine the value of the quantity of interest. I want to figure out which priors were the "best". I'm struggeling with how one should actually define "best" in this context.
Let's illustrate this with some hypothetical priors and a likelihood from the experiment. I.e. I don't get a single correct value, but instead e.g. have an estimate and a standard error. Let's use this plot as an example: It seems relatively clear that we would
- favor priors 4 and 5 over 1 and 2, because they are more certain and not proven wrong by the data, and
- favor priors 1, 2, 4 and 5 over prior 3, because prior 3 has the bulk of the probability mass in what seems to be a region not very compatible with the data.
A metric like the Hellinger distance (between the priors and the normalized likelihood) would capture those judgments of mine. However, it seems like that's not really such a good metric. E.g. it would strongly favor prior 5 over prior 4, which is, not at all, a clear case. This also becomes more obvious when we have less data and e.g. the likelihood is wider (and e.g. looks like prior 2). It seems wrong to me that prior 4 or 5 would become "worse", because we have less data (but it actually does not contradict the priors, at all). Sure, in the case where the likelihood looks more like prior 2, prior 2 should be considered better than under the likelihood shown above in black, but we could not really say that the data contradicts prior 4 or 5, could we?
I suspect someone must have done research on this topic and/or published about it, but somehow I failed to find the right search terms to find how people typically approach this. Any answers on sensible ways of approaching this problem and/or pointers literature would be appreciated.