Tagged Questions

0
votes
0answers
36 views

Identifying argmax result in information theory

I'm reading through a thesis at the moment which uses a "well known result from information theory", namely $\mathrm{argmax}_\mathbf{g}\,\mathbf{f}\,\mathrm{log}\,\mathbf{g} = \mathbf{f}$, for two ...
1
vote
0answers
31 views

Is it possible to calculate mutual information by moments generating functions?

I went to listen to a workshop and some audience asked the presenter how the moments can improve the mutual information. I am learning the MI(Mutual Information) and moments so don't have enough ...
2
votes
0answers
47 views

Multi-information of a uniformly distributed random variable on the L1 sphere

I posted this question in the stackexchange mathematics forum without any reponse. Maybe it was the wrong forum, so I try it here. I tried to compute the multi-information (MI) $I[\mathbf U] = ...
2
votes
0answers
154 views

KL divergence between 2 distributions with unequal cardinalities?

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$. Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or ...
3
votes
2answers
77 views

What are some better ways to encode these symbols?

I was going through some tutorials on information theory. It had the following example concerning transmission of three symbols $A, B$ and $C$ such that $P(A) = 1/3 = P(B) = P(C)$ If we encode the ...
2
votes
0answers
99 views

Normalized mutual information using cardinalities

I have the cardinalities of sets $N_i, \forall i \in 1,2..n$, and the cardinalities $|N_i \cap T|, |N_i|, |T|, |N_i \cup T|, \forall i$, are known. Here, the set $T$ and sets $N_i$'s are all ...
1
vote
2answers
87 views

Mutual information with a Dirac delta type pdf

What does the $MI(X,Y)$ convey about $Y$, when one of the probability distributions, $X$ is trivial and has all the probability concentrated at a single point?
3
votes
1answer
542 views

Kullback-Leibler divergence

Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler ...
1
vote
0answers
110 views

Understanding entropy/salience of probability distribution of two data sets

I'm using a 1.6M tweet corpus to train a naive bayes sentiment engine. I am trying to "compute the entropy of a probability distribution of the appearance of an n-gram in different datasets". My two ...
7
votes
1answer
289 views

Is there any use for the quantity $\int f(x)^2 dx$ in statistics or information theory?

Is there any use for the quantity $$ \int f(x)^2 dx $$ in statistics or information theory?
1
vote
2answers
363 views

Can the mutual information of a “cell” be negative?

Please forgive me if this is not the right Stack Exchange (and for inventing terms). For discrete random variables X and Y, the mutual information of X and Y can be defined as follows: $I(X;Y) = ...