Suppose I have some prior $\pi(\theta)$, from which I draw $N$ samples, each having parameter $\theta_i$. These $\theta_i$'s are known to me. Suppose that one of these samples (unknown to me which) generates some data $y$, following some likelihood $\mathcal{L}(y|\theta)$. Now, I can try to identify which $i$ generated the $y$ by calculating evidences of each $i$ generating the data $y$. The true $i$ will then usually have the highest evidence.

The question that I am interested in is: how predictive is this data $y$ to getting the right $i$? How often can I expect to get it right, and with what certainty? I am essentially asking: what the information content that $y$ gives me for identification?

One way (possibly not a good one of looking at this) is with some information theory-like arguments, so a possible avenue I thought of:

I could calculate the relative entropy (KL-divergence) of the prior to the posterior, $D_{KL}(p(\theta|y)||\pi(\theta))$. This can be converted to a factor, for example, if the information is a factor of 50, I would say that I can expect to identify the right $i$ out of a sample of $N=50$. Is that right? What is the actual relation between accuracy and $N$ given some information as quantified by the relative entropy? Is there a way of quantifying this?

Another way to go about this might be by looking at the distributions of the evidences given a wrong datapoint and a right datapoint, but how to go about this, I am also unsure of.

I feel like stuff this must have been done before, so relevant references would also be appreciated!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.