A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...
0
votes
0answers
15 views
Identifying argmax result in information theory
I'm reading through a thesis at the moment which uses a "well known result from information theory", namely
$\mathrm{argmax}_\mathbf{g}\,\mathbf{f}\,\mathrm{log}\,\mathbf{g} = \mathbf{f}$,
for two ...
0
votes
0answers
31 views
How to prove $H(X-Y)\le \log(2\pi eD)$?
In rate distortion theory, difference error entropy $H(X-Y)\le \log(2\pi eD)$, how can we prove this?
0
votes
1answer
58 views
Reasonable choices of programming languages and the length of program [closed]
I wonder how it's possible that:
it can be shown that all reasonable choices of programming
languages lead to quantification of the amount of absolute information in
individual objects that is ...
1
vote
1answer
36 views
how to calculate E[vech(x x')vech(x x')']?
Supposing a vector x follows normal distribution. I want to calculate the expectation of the "fourth moment" in a vector form, meaning $\text{E}[\text{vech}(x x')\text{vech}(x x')']$, given that we ...
0
votes
0answers
25 views
Different kind of scaling matrices
I was reading through a publication and it suggested there are different kinds of scaling matrices. They suggest the use of the fisher information matrix and I never heard it referred to as a scaling ...
1
vote
0answers
31 views
Is it possible to calculate mutual information by moments generating functions?
I went to listen to a workshop and some audience asked the presenter how the moments can improve the mutual information. I am learning the MI(Mutual Information) and moments so don't have enough ...
1
vote
0answers
32 views
MAE/MSE with or without square root
I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of ...
4
votes
0answers
54 views
Orthogonal intersection in a Riemannian manifold
Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where ...
3
votes
1answer
86 views
Orthogonal intersection of linear family and exponential family
I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question.
In statistical maniolds ...
1
vote
0answers
22 views
Bandits without exploitation: finding the best items with incomplete information
I'm trying to analyze a general game. This is probably well-known, in which case pointers to relevant literature would suffice (but explanation would not be declined!). If it's not standard, of course ...
2
votes
1answer
57 views
Pinsker's Inequality for Bayesian hypothesis testing
I am wondering if one can relate the KL divergence to the probability of error in a Bayesian binary hypothesis testing setting. That is, we have to decide between hypotheses $A$ and $B$ given ...
2
votes
0answers
42 views
Best method for estimating divergence with unequal sample sizes?
Background
I have several datasets of word frequencies where some datasets have much more data than others: from 500 samples to 3000 samples. I also have large reference corpora with millions of ...
2
votes
1answer
76 views
Differential Entropy
Differential entropy of Gaussian R.V. is $\log_2(\sigma \sqrt{2\pi e})$. This is dependent on $\sigma$, which is the standard deviation.
If we normalize the random variable so that it has unit ...
1
vote
0answers
94 views
Mutual Information really invariant to invertible transformations?
"Estimating mutual information" [A Kraskov, H Stögbauer, P Grassberger - Physical Review E, 2004]
states that
Mutual information is invariant under reparametrization of the marginal variables. ...
0
votes
0answers
157 views
What's the correct way to calculate information gain ratio?
I'm trying to implement information gain ratio[1] to find how much a variable
affects contributes to class membership in a naive bayesian classifier.
I hope to use this for both weighting and to find ...
0
votes
0answers
92 views
Similarity order of distributions from Jensen-Shannon divergence
I have n univariate discrete distributions on the same finite domain, and calculated the square-root of the Jensen-Shannon divergence for each pair, yielding a metric distance that is also bounded. Is ...
1
vote
2answers
107 views
How can I use KL-divergence to weight features?
I have a naive Bayes classifier with two classes (target and non-target) and distributions for a number of features (the same for both classes).
I know that some features contribute more, or less to ...
2
votes
1answer
108 views
How, and should I, use KL-divergence to improve a naive Bayes classifier?
How, and should I, use KL-divergence to improve a naive Bayes classifier?
I have a Naive Bayes Classifier working on a number of datatypes (real, boolean and categorical). Each variable is weighted ...
0
votes
0answers
74 views
Is it possible to use Mic (maximal information coefficient) as an 'evolution' of partial correlation?
I have seen on a 11 months ago question here the following quote from a paper referred on the question:
...
2
votes
0answers
47 views
Multi-information of a uniformly distributed random variable on the L1 sphere
I posted this question in the stackexchange mathematics forum without any reponse. Maybe it was the wrong forum, so I try it here.
I tried to compute the multi-information (MI) $I[\mathbf U] = ...
0
votes
0answers
22 views
Multi-stage Filter - how to implement?
I'm reading the paper:
"New Directions in Trac Measurement and Accounting: Focusing on the Elephants, Ignoring the Mice"
I want to experiment with multistage filters. There are some details in the ...
0
votes
0answers
74 views
Conditional Entropy of linear combination of random variables
$X_1 \sim \mathcal{N}(0,P)$, $X_2 \sim \mathcal{N}(0,P)$ and $Z \sim \mathcal{N}(0,N)$ with $X_1,X_2,Z$ mutually independent.
How do we compute
$$
H(aX_1+bX_2+Z \mid \alpha X_1+\beta X_2)
$$
where ...
0
votes
0answers
140 views
Kullback-Leibler divergence for multivariate binomial distributions
I understand KL divergence abstractly, but I'm not exactly sure how you would calculate it for a multivariate binomial distribution (such as an Ising model on a random graph). If I am sampling 100 ...
3
votes
1answer
130 views
ID3 and C4.5: how does “gain ratio” normalize “gain”?
The ID3 algorithm uses "Information Gain" measure.
The C4.5 uses "Gain Ratio" measure which is Information Gain divided by SplitInfo, whereas SplitInfo is high for a split where records split evenly ...
1
vote
0answers
119 views
Conditional entropy of sum of random variables
How can be proven that for random variables $A$ and $B$, and $C = A + B$,
$$H(C\mid A) = H(B\mid A).$$
Also, would it be possible to determine if $H(C)$ would be greater than $H(A)$?
4
votes
2answers
105 views
Number of needed samples for entropy estimation
I would like to estimate the entropy of a source generating binary vectors of length M which are very sparse (just a few 1s), using the naive (empirical) estimator ...
2
votes
0answers
153 views
KL divergence between 2 distributions with unequal cardinalities?
Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$.
Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or ...
3
votes
0answers
113 views
Calculating the Fisher Information of bivariate normal
I'm lost. If I estimated the a Gaussian mixture model, with a shared diagonal covariance, will the Fisher information of the means be $\Sigma^{-1}$ ?
7
votes
2answers
499 views
Feature selection using mutual information in Matlab
I am trying to apply the idea of mutual information to feature selection, as described in these lecture notes (on page 5).
My platform is Matlab. One problem I find when computing mutual information ...
3
votes
2answers
77 views
What are some better ways to encode these symbols?
I was going through some tutorials on information theory. It had the following example concerning transmission of three symbols $A, B$ and $C$ such that
$P(A) = 1/3 = P(B) = P(C)$
If we encode the ...
2
votes
0answers
98 views
What is Shannon's source entropy?
Suppose that ${X_n; Y_n}$ is a random process with a discrete alphabet, that is, taking on values in a discrete set for $n$ data length. They correspond to the input and output of a communication ...
3
votes
0answers
79 views
Mutual information in a mixture of independent variables
Suppose we have a $n$ pairs of mutually independent variables over $k$ outcomes and take a mixture distribution, what can we say about mutual information in the mixture? In particular I'm wondering ...
1
vote
0answers
80 views
How to combine different feature values to build a score for ranking
I am doing some information extraction study on some data. We extract a "word" list from the data. And for each "word", it has a n dimensional feature vector in R ^n. Next, We want to rank the list ...
1
vote
0answers
46 views
Conditionalising increases expected information?
I saw a talk that mentioned this theorem. I am now trying to find details of the theorem, but I'm not having any luck. I hope that if I can describe roughly what the theorem said, someone can ...
2
votes
0answers
117 views
Calculating entropy of a binary matrix
Given a matrix whose entries consist of only 1's and 0's, I would like to come up with a measure of how "ordered" the matrix is, in some sense. This exact question was posed here:
Measuring entropy/ ...
4
votes
2answers
182 views
Definition and origin of “cross entropy”
Without citing sources, Wikipedia defines the cross-entropy of discrete distributions $P$ and $Q$ to be
\begin{align}
\mathrm{H}^{\times}(P; Q)
&= -\sum_x p(x)\, \log q(x).
\end{align}
...
1
vote
0answers
58 views
GWAS and Statistical theory - does the likelihood of a detectable main effect decrease with complexity?
I've been wondering recently about the difficulty of detecting statistically significant results from, say, genome-wide association studies. In these studies, many - ($10^{8}$) for example- ...
5
votes
3answers
215 views
Information theoretic central limit theorem
The simplest form of the information theoretic CLT is the following:
Let $X_1, X_2,\dots$ be iid with mean $0$ and variance $1$. Let $f_n$ be the density of the normalized sum $\frac{\sum_{i=1}^n ...
5
votes
1answer
704 views
Jensen-Shannon divergence calculation for 3 prob distributions: Is this ok?
I would like to calculate the jensen-shannon divergence for he following 3 distributions. Is the calculation below correct? (I followed the JSD formula from wikipedia):
...
3
votes
0answers
42 views
Discerning the best model for a problem
This is a vague question. I will do my best, I think it has definite answers. I am hoping for answers of the form "Read book x, learn this specific topic, read this paper/s".
What is bothering me is ...
5
votes
2answers
554 views
What is empirical entropy?
In the definition of jointly typical sets (in "Elements of Information Theory", ch. 7.6, p. 195), we use
$$-\frac{1}{n} \log{p(x^n)}$$ as the empirical entropy of an $n$-sequence with $p(x^n) = ...
1
vote
1answer
158 views
Feature selection weighting 2 filters in Naive Bayes
I am trying to do text classification using Naive Bayes. Before training, I would like to make feature selection in order to reduce the feature space dimension. In order to do so, I have thought of ...
2
votes
0answers
99 views
Normalized mutual information using cardinalities
I have the cardinalities of sets $N_i, \forall i \in 1,2..n$, and the cardinalities $|N_i \cap T|, |N_i|, |T|, |N_i \cup T|, \forall i$, are known.
Here, the set $T$ and sets $N_i$'s are all ...
1
vote
2answers
87 views
Mutual information with a Dirac delta type pdf
What does the $MI(X,Y)$ convey about $Y$, when one of the probability distributions, $X$ is trivial and has all the probability concentrated at a single point?
1
vote
0answers
52 views
Tractability of mutual information-augmented ensemble classification algorithms
I am seeking to augment random forest classification using Shannon-Weaver mutual information as a metaheuristic to partition candidate datasets. Specifically, I am trying to determine if such an ...
7
votes
2answers
214 views
Results on Monte Carlo estimates produced by importance sampling
I have been working on importance sampling fairly closely for the past year and have a few open-ended questions that I was hoping to get some help with.
My practical experience with importance ...
3
votes
1answer
540 views
Kullback-Leibler divergence
Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal
$p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler
...
0
votes
1answer
155 views
Measuring what's 'lost' in PCA dimensionality reduction?
I'm working with multiple time series signals ${\{X_i\}}$ and one method to remove suspected noise is with PCA. But unlike most methods which remove the components with least variance, we remove much ...
3
votes
1answer
150 views
Property of KL-divergence
Let $p_1$ and $p_2$ be two distinct probability distributions. Define
$$
L(q)=D(q||p_1)-D(q||p_2)
$$
where $D$ is the usual Kullback-Leibler divergence. Assume the support of $p_2$ is included in ...
3
votes
1answer
378 views
Can mutual information gain value be greater than 1
I have a very basic doubt. Sorry if this irritates few. I know that Mutual Information value should be greater than 0, but should it be less than 1 ? Is it bounded by any upper value ?
Thanks,
Amit.
