# Tagged Questions

A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

5 views

### Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
22 views

### Computing the Interaction gain. Is there an Error in the infotheo package in R?

In order to implementing a certain feature selection method for a classification problem I need to estimate the the interaction the interaction gain between two features and the target variable which ...
214 views

### Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
23 views

### Is there an intuitive interpretation of mutual information values (bits, nits)?

I understand how mutual information is calculated, and what it is addressing: how much the distribution of one variable changes conditional on the value of another variable. But I don't really ...
45 views

### References that justify use of Gaussian Mixtures

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much ...
11 views

### optimization problem rewrite help

From my understanding, the transformation between probability and information content for d=2 is as follows: Information = -log_2 (Probability) If I have an optimization problem that is written in ...
64 views

### What makes something a “Probability”?

What makes it legitimate to say a set of values are probabilities? (I need an answer for a formalisation reasons.) Let me give a simple example. Let's assume that I have a number of letters and each ...
257 views

### Different AIC definitions

From Wikipedia there is a definition of Akaike's Information Criterion (AIC) as $AIC = 2k -2 \log L$, where $k$ is the number of parameters and $\log L$ is the log-likelihood of the model. However, ...
60 views

### Find the input/output probability distributions that realize the capacity of communications channel?

Consider two random variables $X,Y$. For simplicity, they are discrete and finite. Let $Q(y|x)$ be the conditional probability of $Y$ given $X$. Their mutual information is defined as: ...
29 views

### Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
28 views

### “Pairwise dependence probability”

From pg 9 of [1]: The notion of dependence between two variables A and B is taken to be mutual information; the amount of evidence for dependence is then the probability that the mutual ...
14 views

### When can conditional mutual information be decomposed as a sum?

Let $X$, $Y$, $Z$ be discrete random variables, each on his own support. What are the necessary conditions to be able to write the following: $$I(X;Y|Z) = \sum_z p(z) \cdot I(X;Y|Z=z)$$ Isn't this ...
65 views

### Expected and observed Fisher information?

Studying asymptotics, I bumped into the concept of Observed Fisher Information, as a way to compute Fisher Information when the parameter $\theta$ is unknown. I am also aware that it is related in ...