15
$\begingroup$

Could the mutual information over the joint entropy: $$ 0 \leq \frac{I(X,Y)}{H(X,Y)} \leq 1$$

be defined as:"The probability of conveying a piece of information from X to Y"?

I am sorry for being so naive, but I have never studied information theory, and I am trying just to understand some concepts of that.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Welcome to CV, luca maggi! What a lovely first question! $\endgroup$
    – Alexis
    Commented Jan 17, 2019 at 19:56

3 Answers 3

Reset to default
11
$\begingroup$

The measure you are describing is called Information Quality Ratio [IQR] (Wijaya, Sarno and Zulaika, 2017). IQR is mutual information $I(X,Y)$ divided by "total uncertainty" (joint entropy) $H(X,Y)$ (image source: Wijaya, Sarno and Zulaika, 2017).

enter image description here

As described by Wijaya, Sarno and Zulaika (2017),

the range of IQR is $[0,1]$. The biggest value (IQR=1) can be reached if DWT can perfectly reconstruct a signal without losing of information. Otherwise, the lowest value (IQR=0) means MWT is not compatible with an original signal. In the other words, a reconstructed signal with particular MWT cannot keep essential information and totally different with original signal characteristics.

You can interpret it as probability that signal will be perfectly reconstructed without losing of information. Notice that such interpretation is closer to subjectivist interpretation of probability, then to traditional, frequentist interpretation.

It is a probability for a binary event (reconstructing information vs not), where IQR=1 means that we believe the reconstructed information to be trustworthy, and IQR=0 means that opposite. It shares all the properties for probabilities of binary events. Moreover, entropies share a number of other properties with probabilities (e.g. definition of conditional entropies, independence etc). So it looks like a probability and quacks like it.

Wijaya, D.R., Sarno, R., & Zulaika, E. (2017). Information Quality Ratio as a novel metric for mother wavelet selection. Chemometrics and Intelligent Laboratory Systems, 160, 59-71.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ How is IQR function defined for $A\subset\Omega$ in order to check against the defining properties of the probability measure? Are you introducing $I(X',Y')$ and $H(X',Y')$ with $X':=XI(A),\, Y':=YI(A)$ where $I$ is the characteristic function? $\endgroup$
    – Hans
    Commented Jan 14, 2019 at 3:35
  • $\begingroup$ Well, my question is directed at a part of your answer and not a stand alone question. Are you suggesting that I open a new question and link and direct it to your answer? $\endgroup$
    – Hans
    Commented Jan 14, 2019 at 9:31
  • $\begingroup$ @Hans What I said, is that this measure easily fits the definition, correct me if I'm wrong. Axioms 1. and 2. are obvious. For axiom 3., $I(X, Y)$ is the overlap, $H(X, Y)$ is the total space, so the fraction can be easily seen as probability. $\endgroup$
    – Tim
    Commented Jan 14, 2019 at 10:24
  • 1
    $\begingroup$ A probability is defined on a sample space and its sigma field $(\Omega, \mathscr{F})$. I am confused as to what these are for this probability measure IQR. There is already a sample space and its sigma field for the probability measure defined for the random variables $X$ and $Y$. Is the sample space and field of the new probability measure IQR the same as those of the old probability measure associated with $X$ and $Y$? If not, how are they defined? Or, are you saying these need not be defined? How then do you check it against the axioms? $\endgroup$
    – Hans
    Commented Jan 14, 2019 at 19:07
  • $\begingroup$ @Hans I stated it explicitly that this is consistent with axioms, but it is hard to say probability of what exactly this would be. The interpretation I suggested is probably of reconstructing signal. This is not a probability distribution of X or Y. I guess you could go deeper into interpreting and understanding it. The question was if this could be interpreted as probability and answer was that formally yes. $\endgroup$
    – Tim
    Commented Jan 14, 2019 at 19:42
2
$\begingroup$

Here is the definition of a probability space. Let us use the notations there. IQR is a function of a tuple $(\Omega,\mathscr F,P,X,Y)$ (The first three components form the probability space the two random variables are defined on). A probability measure has to be a set function that satisfy all the conditions of the definition listed in Tim's answer. One will have to specify $\Theta:=(\Omega,\mathscr F,P,X,Y)$ as some subset of a set $\tilde\Omega$. Moreover, the set of $\Theta$'s has to form a field of subsets of $\tilde\Omega$, and that $\text{IQR}(\Omega,\mathscr F,P,X,Y)$ has to satisfy all three properties listed in the definition of probability measure listed in Tim's answer. Until one constructs such an object, it is wrong to say IQR is a probability measure. I for one do not see the utility of such a complicated probability measure (not the IQR function itself but as a probability measure). IQR in the paper cited in Tim's answer is not called or used as probability but as a metric (The former is one kind of the latter, but the latter is not one kind of the former.).

On the other hand, there is a trivial construction that allows any number on $[0,1]$ to be a probability. Specifically in our case, consider any given $\Theta$. Pick a two-element set as the sample space $\tilde\Omega:=\{a,b\}$, let the field be $\tilde{\mathscr F}:=2^{\tilde\Omega}$ and set the probability measure $\tilde P(a):=\text{IQR}(\Theta)$. We have a class of probability spaces indexed by $\Theta$.

Improve this answer
$\endgroup$
6
  • $\begingroup$ For your information, I edited my answer to simplify & clarify it. Probability is a metric that has some special properties. We are talking about set of all the possible pairs of messages and their reconstructions $(x_i, y_i)$. Here the random variable is a complicated, unknown function that tells us if the reconstruction was "good" or not. Returning trustworthy reconstruction can be thought as a binary event, my answer is simply that IQR can be thought as probability for such event (or rather approximation of it). $\endgroup$
    – Tim
    Commented Jan 17, 2019 at 10:56
  • $\begingroup$ @Tim: The previous version of your answer is a much better answer as it provides a clear definition one can check against. There is no way to circumvent a definition. Probability is a metric, but not all metric with "some special properties" is a probability. Until we can verify all the "special properties" of this metric fit the defintion, it is not one. However, I did give a trivial construction of a class of probability spaces indexed by the parameter tuple $\Theta:=(\Omega,\mathscr F,P,X,Y)$. $\endgroup$
    – Hans
    Commented Jan 18, 2019 at 2:49
  • $\begingroup$ That is also the case if you use complicated neural network with sigmoid activation function at the end, can you prove that the output is probability in metric-theoretical terms..? Yet, we often choose to interpret this as probability. $\endgroup$
    – Tim
    Commented Jan 18, 2019 at 10:09
  • $\begingroup$ @Tim: Of course you can. That is an easy one to deal with using the pullback measure. The sigmoid function is a measurable function which already stipulates the sigma fields of the domain and range ($[0,1]$ with the (conventional) Borel field) of the function. The probability measure of a subset $A$ of the sample space $P(A):=\mu(f(A))$ where $\mu$ is the (conventional) Borel measure of $R$ and $f$ is the sigmoid function. QED $\endgroup$
    – Hans
    Commented Jan 18, 2019 at 20:50
  • 1
    $\begingroup$ Sorry, but I never found this kind of discussions & measure theory interesting, so I'll withdraw from the further discussion. I also do not see your point in here, especially since your last paragraph seems to say exactly the same thing I was saying from the very begging. $\endgroup$
    – Tim
    Commented Jan 18, 2019 at 21:07
2
$\begingroup$

Going back in history a bit, the role of $\frac{I(X,Y)}{H(X,Y)} $ as a measure of probability can be seen, in part, in the 1961 article by Rajski: A Metric Space of Discrete Probability Distributions. This article outlines the development of the Rajski Distance ${(D_R)}$ is:

$${D_R}=1 - \frac{I(X,Y)}{H(X,Y)} $$

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Interesting that it is a metric. +1. $\endgroup$
    – Hans
    Commented Feb 26, 2022 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.