Skip to main content
Cross Validated

Return to Answer

typos...
Source Link
Yair Daon
Yair Daon
  • 2.7k
  • 1
  • 19
  • 33

Since I don't understand what you need exactly with mutual information, let me suggest a different approach.

It seems to me that you are trying to compare the performance of two algorithms $A, B$, while you believe that $B$ is, in fact, true. It might be a good idea, then, to just use $B$ all the time then. But, maybe you have some restriction that prevents you from using it (e.g. it is too computationally demanding). So you might want to consider a "cheaper" algorithm. In this case, since you are already in the realm of information theory - why not use the $KL$ divergence?

You know the distribution $p_{B}(X)$:

\begin{align} p_B(X_1 = 1) = 0.4 &\text{ , } p_B(X_1 = 0) = 0.6\\ p_B(X_2 = 1) = 0.35 &\text{ , } p_B(X_2 = 0) = 0.55\\ p_B(X_3 = 1) = 0.8 &\text{ , } p_B(X_3 = 0) = 0.2\\ \end{align}

and you also have $p_A(X)$. To compare them, you may calculate $\mathbb{KL}(p_B || p_A)$. The interpretation is that this is the information lost when approximating $p_B$ (which is the truth!!) using $p_A$ (which might be easier to calculate).

Since I don't understand what you need exactly with mutual information, let me suggest a different approach.

It seems to me that you are trying to compare the performance of two algorithms $A, B$, while you believe that $B$ is, in fact, true. It might be a good idea, then, to just use $B$ all the time then. But, maybe you have some restriction that prevents you from using it (e.g. it is too computationally demanding). So you might want to consider a "cheaper" algorithm. In this case, since you are already in the realm of information theory - why not use the $KL$ divergence?

You know the distribution $p_{B}(X)$:

\begin{align} p_B(X_1 = 1) = 0.4 &\text{ , } p_B(X_1 = 0) = 0.6\\ p_B(X_2 = 1) = 0.35 &\text{ , } p_B(X_2 = 0) = 0.55\\ p_B(X_3 = 1) = 0.8 &\text{ , } p_B(X_3 = 0) = 0.2\\ \end{align}

and you also have $p_A(X)$. To compare them, you may calculate $\mathbb{KL}(p_B || p_A)$. The interpretation is that this is the information lost when approximating $p_B$ (which is the truth!!) using $p_A$ (which might be easier to calculate.

Since I don't understand what you need exactly with mutual information, let me suggest a different approach.

It seems to me that you are trying to compare the performance of two algorithms $A, B$, while you believe that $B$ is, in fact, true. It might be a good idea, then, to just use $B$ all the time. But, maybe you have some restriction that prevents you from using it (e.g. it is too computationally demanding). So you might want to consider a "cheaper" algorithm. In this case, since you are already in the realm of information theory - why not use the $KL$ divergence?

You know the distribution $p_{B}(X)$:

\begin{align} p_B(X_1 = 1) = 0.4 &\text{ , } p_B(X_1 = 0) = 0.6\\ p_B(X_2 = 1) = 0.35 &\text{ , } p_B(X_2 = 0) = 0.55\\ p_B(X_3 = 1) = 0.8 &\text{ , } p_B(X_3 = 0) = 0.2\\ \end{align}

and you also have $p_A(X)$. To compare them, you may calculate $\mathbb{KL}(p_B || p_A)$. The interpretation is that this is the information lost when approximating $p_B$ (which is the truth!!) using $p_A$ (which might be easier to calculate).

Source Link
Yair Daon
Yair Daon
  • 2.7k
  • 1
  • 19
  • 33

Since I don't understand what you need exactly with mutual information, let me suggest a different approach.

It seems to me that you are trying to compare the performance of two algorithms $A, B$, while you believe that $B$ is, in fact, true. It might be a good idea, then, to just use $B$ all the time then. But, maybe you have some restriction that prevents you from using it (e.g. it is too computationally demanding). So you might want to consider a "cheaper" algorithm. In this case, since you are already in the realm of information theory - why not use the $KL$ divergence?

You know the distribution $p_{B}(X)$:

\begin{align} p_B(X_1 = 1) = 0.4 &\text{ , } p_B(X_1 = 0) = 0.6\\ p_B(X_2 = 1) = 0.35 &\text{ , } p_B(X_2 = 0) = 0.55\\ p_B(X_3 = 1) = 0.8 &\text{ , } p_B(X_3 = 0) = 0.2\\ \end{align}

and you also have $p_A(X)$. To compare them, you may calculate $\mathbb{KL}(p_B || p_A)$. The interpretation is that this is the information lost when approximating $p_B$ (which is the truth!!) using $p_A$ (which might be easier to calculate.