Tagged Questions

A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

15 views

Is it valid to reduce the AICc penalty for multiple variables, when some variables should be grouped?

I have a data set of mean trait values for each of 18 populations, and want to test whether several ecological variables are related to variation in traits. I'm using the corrected Akaike information ...
12 views

How to deal with missing data when calculating Information Gain

While working on a neural network for classification problem I'm dealing with huge number of possible features and information gain seems like a good way to narrow them down (there are hundreds of ...
27 views

In feature selection, are there any rules on choosing metrics to mesure relevance? (MI / Fisher score / correlation coefficient, etc)

This is a rather general question. If the question is vague and hard to answer in a few lines, I'd be happy if someone just point me to some readings. Thanks in Advance. I am working on a multi-class ...
36 views

How to calculate the information loss of PCA?

How would we calculate the information loss of reducing dimensions using PCA ? Would it be the amount of variance loss if we skip certain eigenvectors after the PCA ?
55 views

What does the Akaike Information Criterion (AIC) score of a model mean?

I have seen some questions here about what it means in layman terms, but these are too layman for for my purpose here. I am trying to mathematically understand what does the AIC score mean. But at ...
364 views

Interpretation of the entropy with a coding length?

There is this interpretation of the entropy $-\sum_i p_i \log_2 p_i$ as the average length (in bits) per character when using an optimal encoding of a message. Now, if we use the simple 3-letter case ...
42 views

Transfer entropy value between 0 and 1

Given two variables, X and Y, there is a way of obtaining a Mutual Information value between 0 and 1 by: MI_normalised=MI_original/sqrt(H(X)*H(Y)); where H(X) ...
9 views

Is Lehmann-informativeness transitive?

Experiments $\mathbf{E}$ and $\mathbf{F}$ are defines as follows: An experiment $\mathbf{E}$ is a random quantity $X$ and a family $\mathbf{P} = \{P_\theta,\,\theta\in\Omega\}$ of possible ...
31 views

20 views

What is the best way to decide bin size for computing Entropy or Mutual Information?

I have a continuous distribution that I was thinking of binning for computing MI and H. I often arbitrarily decide on bin size. Is there a general consensus on how to set bin size and number? Thanks ...
39 views

35 views

Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
29 views

“Pairwise dependence probability”

From pg 9 of [1]: The notion of dependence between two variables A and B is taken to be mutual information; the amount of evidence for dependence is then the probability that the mutual ...
20 views

When can conditional mutual information be decomposed as a sum?

Let $X$, $Y$, $Z$ be discrete random variables, each on his own support. What are the necessary conditions to be able to write the following: $$I(X;Y|Z) = \sum_z p(z) \cdot I(X;Y|Z=z)$$ Isn't this ...
112 views

Expected and observed Fisher information?

Studying asymptotics, I bumped into the concept of Observed Fisher Information, as a way to compute Fisher Information when the parameter $\theta$ is unknown. I am also aware that it is related in ...