Suppose there is some distribution $F$ not known to us. However, we can get information about this distribution by means of samples, i.e. we have a set of random variables from this distribution. Ideally, I would like to quantify information I have about this distribution.
For example, if we have a coin with bias $p$, then after (sufficiently large) $N$ tosses we would have much lesser uncertainty about the value of $p$ than say if we had access to only one toss. I want to quantify this uncertainty.
During my general search for a related concept, I came across the idea of "entropy" in information theory. (btw, I did little bit of information theory in school and since then I have not used it once). This is very similar to what I want to do but entropy is defined for the source... or a single random variable. I could not find anything for a set of random variables.
