Overview

I want to perform a Bayesian model selection on many datasets and use these datasets to determine the required parameter priors.

Example Scenario: Coins

Suppose I have a collection of thousand coins produced by a machine that randomly produces fair and loaded coins. The loaded coins are not identical, but their heads ratio $θ$ follows an unknown distribution $p(θ|\mathcal{M}_\text{loaded})$ obeying some constraints (see below). For each coin, I want to decide whether it’s fair using Bayesian model selection with two models $\mathcal{M}_\text{loaded}$ and $\mathcal{M}_\text{fair}$.

I know:

• For each coin: the number of heads from hundred tosses (and thus an estimator $\hat{θ}$ for the heads ratio $θ$).
• Model priors $p_\text{fair}$ and $p_\text{loaded}$ with $0.1≤p_\text{fair}≤0.9$.
• The probability density $p(θ|\mathcal{M}_\text{loaded})$ of the heads ratio of the loaded coins obeys the following constraints:
  • symmetric around ½,
  • smooth,
  • not very far from a uniform distribution, say, $0.1 < p(θ|\mathcal{M}_\text{loaded}) < 10$ everywhere.

With all this given, the main information I am lacking for this is a prior for $p(θ|\mathcal{M}_\text{loaded})$. I estimate this by finding a suitable distribution and fitting it to my data for all coins, ignoring coins with $0.4<\hat{θ}<0.6$, since those have a decent chance to be fair. The rest of the Bayesian model selection is straightforward.

Questions

• Is this procedure sound? I acknowledge that I use the same data twice. However, the data for a given coin has barely any impact on the parameter priors relevant to its model selection. (I could also exclude the data for the given coin when determining the priors for its analysis, doing a thousand fits instead of just one.)

• If yes, is there a name or reference for this approach?

• If no, is there a better way to determine parameter priors for $\mathcal{M}_\text{loaded}$? I am particularly interested in ways that can be extended to a more complex model space as well as higher-dimensional and unbounded parameter spaces.