I am currently thinking about the following problem. Suppose you have a simple Naive Bayes model for binary classification based on binary random variables. For example, suppose you want to predict whether a patient has a certain diagnosis or not (denoted by the random variable $Y$) based on some binary medical test observations (denoted by the random variables $X_1, \ldots, X_N$). Suppose also that we use Beta conjugate priors on the binomial variables. Depending on how much data you have actually collected, the density of the beta distribution for the prior may change. Observing more samples makes the density function more spiky.

Now, one way how to quantify the effectiveness / informativeness of each medical test is to measure the conditional mutual information $I(Y | X_i) = H(Y) - H(Y|X)$ which in this case would mean:"How much entropy does the $X_i$ test reduce from the diagnosis variable?"

My question is the following: "Is there any known way how to bound the mutual information computation given the counts on the beta priors for both $X$ and $Y$?". Intuitively, the "denser" your prior, the tighter should be the bounds on mutual information. However, I am still puzzled whether this is a problem that has been well-studied already or not.

What makes this problem difficult is the fact that the mutual information is a non-monotone function over $p(Y=y)$ and $P(X=x|Y=y)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.