This is a problem I have encountered while considering ELO ratings for a game like Go.
Suppose we have two players, one has a known rating, the other's is unknown but assumed to be close to the first. We match the players to gain a more accurate estimate for the latter's rating.
The player has about a 50% chance to win, so I want to be able to say that we gain 1 bit of information from the result of the game (and I intend to follow with discussing ways to obtain more than 1 bit from a game, e.g. by considering the score difference rather than just a boolean win/lose).
But if taken literally, it gives nonsensical results - for example, it would mean that after 64 games, we can tell the player's rating to a precision of 64 bits.
Generally speaking, the sd of the posterior distribution for the rating goes down by the square root of the number of games played, not exponentially as can be implied by the "1 bit per game" statement.
So I thought maybe I should consider this more carefully and look at the entropy of the rating's prior distribution, and see how it differs from the posterior's. I thought that maybe the extra bit of information will manifest as a difference of 1 bit in the entropy of the distributions.
But it's not even close. For a simpler example I looked at the well-studied problem of a coin with unknown bias. If our prior distribution for the probability of Heads is Beta (100, 100), this has an entropy of -2.77846 bits (unless I made a mistake). After one toss, the posterior will be either Beta (101, 100) or Beta (100, 101), each of which has an entropy of -2.78206 bits. So we've gained only about 0.0036 bits, and not 1 as hoped.
So my question is threefold -
If "1 bit" is the correct amount of information gained from the sample, what metric can be used to encode how much information we have accumulated?
If 0.0036 bits is the correct amount of information gained in the coin toss example above, what is a simple way to determine this gain, despite the intuitive argument for a gain of 1 bit?
What else am I missing here?
Edit: To clarify what it is I'm trying to do, let's say there is an unknown quantity $X$ that I want to determine. I have a prior distribution $P(X)$. I draw a sample $Y$ from a known family of distributions parameterized by $X$, $Y \sim P_X(Y)$. However, $Y$ itself is unknown to me - I only know $Z=f(Y)$ for some known function $f$ (of course, it is possible that we have the special case $Z=Y$).
I want to use my knowledge of $Z$ to refine my estimation for $X$. I want to quantify how much observing $Z$ has helped me.
My original approach was to look at the entropy of the distribution $P(Z)$. This depends on the function $f$ - if it is injective, by knowing $Z$ we actually know $Y$, so we gain the maximum information. Accordingly, $P(Z)$ is supported on many different values and has high entropy.
On the other extreme, if $f$ maps all values of $Y$ to a single value, $P(Z)$ is supported on a singleton and has 0 entropy, and we have gained no information.
In between, the more often $f$ maps different values of $Y$ to the same value of $Z$, the lower is the entropy of $P(Z)$, and the less I can deduce about $X$.
So I'm trying to figure out what measure of the uncertainty on $X$ will decrease in accordance to the entropy of $P(Z)$. Or, if not specifically to the entropy, to some other attribute of the distribution of $P(Z)$. This way, I can compare different functions $f$ and $g$ and say that one is better than the other, because one of them gives $a$ units of information per sample, and the other $b$ units.
In the Go game example, $f$ could give a boolean result of who won, and $g$ could give the actual score. In this case, $g$ would be more informative.
Edit 2: I think it's worth emphasizing that this is a rather open and broad question. The exact approach I originally tried obviously doesn't work (and Bridgeburners has shed some light on the underlying reasons); and it's probably impossible to construct something that matches my expectations fully (e.g., there are examples where we can have $Z$ grow arbitrarily in entropy, with a finite bound on the knowledge gained on the distribution of $X$.
I'm basically fishing for what it is that can be said along the goal I outlined. It will probably take the form of a one-sided bound. That is, given the entropy of $P(Z)$, there is likely no lower bound on the knowledge gained on $X$ ($Z$ could be completely random and irrelevant), but there should be an upper bound - if the entropy of $P(Z)$ is $a$, then it can't possibly improve our knowledge on $X$ by more than this-and-that. But I don't know what measure on the uncertainty on $X$ lends itself to such a bound (the only measures I can think of are the sd of the distribution of $X$ and its differential entropy), or what the bound could be.
It's possible that "1 divided by the variance of the distribution of $X$" is what I'm looking for, as it increases linearly with the number of samples, and conceivably, an experiment with 2 bits of entropy for the result should increase it by twice as much as an experiment with 1 bit. But I'm not sure how to quantify the relation between the two.