Questions tagged [information-theory]
A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in predicting a random variable.
678
questions
0
votes
0
answers
11
views
Generalised Jensen-Shannon divergence - What is a small JSD?
I am comparing the similarity between multiple distributions based on the output of different machine-learning models. I am applying the generalised JS divergence (wiki):
$$
JSD_{\pi_1,...,\pi_n}(p_1,....
- 165
3
votes
1
answer
43
views
Is it true that finite expectation leads to finite entropy?
Let $X$ be a random variable supported by the naturals, with a finite expectation. Show that $H(x)<\infty$. Also, give an example of a random variable supported by the naturals that has $H(x)=\...
- 41
0
votes
0
answers
20
views
Understanding Mutual Information as a Measure of Relationship: Why is Mutual Information Different for Perfectly Correlated Deterministic Functions?
Can someone explain why the mutual information (MI) between a1 and a2 is smaller than the MI between ...
- 412
1
vote
1
answer
26
views
How to generalise non-asymptotic Cramer-Rao lower bound for unbiased estimators in semiparametric models?
We all know the classic Cramer-Rao bound which specifies a lower bound of any unbiased estimator's variance in a parametric model. Note that this bound is non-asymptotic in a sense that it is valid ...
- 11
1
vote
0
answers
52
views
Show that conditional residual extropy $r(X,Y)=0$ when random variable $Y$ is a function of $X$
Let $X$ and $Y$ be two random variables. Then a measure called conditional residual extropy, defined as
$$J(X|Y)=-\frac{1}{2}\int_0^\infty \bar{F}^2(x|y)dx, \hspace{2mm}y>0$$ where $\bar{F}_{X|Y}(x\...
- 43
0
votes
0
answers
14
views
Mutual Information of nonadjacent nodes in Bayesian Network
How do you compute the mutual information of two non-adjacent nodes in a Bayesian network?
In this case, what would $I(D;A)$ be? Would I need to take the conditional probabilities of all intemediate ...
- 35
2
votes
1
answer
38
views
differential entropy for comparison distributions
I want to use differential entropy to compare the outcome of Bayesian updating (multidimensional probability distributions) for different datasets. My parameters are different physical parameters i.e. ...
- 61
2
votes
3
answers
117
views
Is feature-extraction and dimensionality-reduction a kind of compression?
I'm struggling to understand what these terms have in common:
Feature extraction
Feature selection
Compression
Dimensionality reduction
Relatedly, the information / entropy in our data should always ...
- 23
1
vote
0
answers
27
views
Fourier transform in information transfer in biological neural network
Principles of Neural Design by Peter Sterling and Simon Laughlin describes a usage of information theory in calculating the rate of information transfer in the brain.
...when successive signal states ...
- 11
0
votes
0
answers
4
views
Sample Complexity of BHT with varying degrees of (large) compression
In Communication-constrained hypothesis testing:
Optimality, robustness, and reverse data processing inequalities, the following (up to some mild editing to highlight my question) is established.
...
- 173
0
votes
0
answers
20
views
Interpretation of time series spectral entropy values wrt forecastability by a general neural network
I recently started using spectral entropy to analyze time series (already windowed). I'm having difficulty for interpreting the results, the entropy of the last 25% of a series is 0.19, and the ...
- 51
2
votes
0
answers
47
views
Shannon source coding theorem and differential entropy
Loosely speaking, Shannon's source encoding theorem says that there is an encoder with rate at least $H(x)$ such that $n$ repetitions of the source can be mapped to at least $nH(X)$ bits of binary ...
- 121
2
votes
1
answer
75
views
Chain rule conditional entropy
A textbook I am reading states that$$H(X,Y)=H(X)+H(Y|X)$$where $H(X,Y)$ is the joint entropy of random variables $X,Y$, $H(X)$ the entropy of $X$, and $H(Y|X)$ is conditional entropy. It then states ...
- 135
0
votes
0
answers
22
views
How do you choose to put a distribution on the right or left of KL divergence? [duplicate]
I always thought of KL divergence as a distance metric between distributions, much like Earth-Movers distance. But I can no longer ignore the asymmetry. A real distance metric is symmetric.
How should ...
- 441
1
vote
0
answers
29
views
Mutual Information decay
Consider $m$ channels indexed by $i$ with $1 \leq i \leq m$. The input alphabets are from the same finite set $\mathcal{X}$. Let $\pi$ denote a probability distribution on $\mathcal{X}$. Define the ...
- 155
3
votes
0
answers
289
views
Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation
TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this.
This is how I understand Minimum ...
- 187
0
votes
1
answer
44
views
Is there any work linking information channel theory to statistical inference?
I wonder what is the theoretical limit of a statistical inference problem. For example we have a model with many parameters, and we can sample many data points from the model. This can be viewed as a ...
1
vote
0
answers
33
views
Choosing number of lag AND model form for Augmented Dickey-Fuller test
Before realising an Augmented Dickey-Fuller (ADF) test, one has to answer 2 questions, how many lags p to include in the model, AND which model to choose among the following:
No constant, no trend
...
- 11
1
vote
1
answer
49
views
Minimum entropy decomposition of probability distributions
Say you want to decompose a probability distribution (a PDF) into a mixture of distributions in such a way as to minimize the mean entropy of the component distributions. I have an idea that this is ...
- 11
1
vote
0
answers
17
views
Effect on entropy when we scale Bernoulli plus Gaussian
Question: Given $X\sim\text{Bernoulli}(\alpha)$, $Y\sim\mathcal{N}(0,1)$, and non-random positive constants $C,\epsilon>0$. Let $H(\cdot)$ be the differential entropy. Is it true that
$$
H((C+\...
- 229
0
votes
0
answers
47
views
KL divergence between normal and skewnormal distribution
I am trying to find an analytical expression for the KL divergence between a normal distribution and a skewnormal distribution. In this paper https://www.mdpi.com/1099-4300/14/9/1606 they derive the ...
- 1
1
vote
2
answers
96
views
Why do information criteria provide consistent estimation of the lag order?
Consider the following $AR(p)$ model $$Y_{t}=\sum_{j=1}^{p}\phi_{j}Y_{t-j}+\epsilon_{t},$$ where $\epsilon_{t}\sim i.i.d\ \mathcal{N}(0,\sigma^{2}).$ Say we have samples $t=1,\dots, T$. The true lag ...
- 123
0
votes
0
answers
39
views
Under What Conditions Does a Gaussian Mixture Model (GMM) Have Maximum Entropy?
Introduction
I'm delving into Gaussian Mixture Models (GMMs) within unsupervised learning frameworks and am particularly interested in their statistical properties, with a focus on entropy. Entropy ...
- 113
0
votes
1
answer
47
views
Mutual information with convex combination of dependent variable and random noise
Suppose I have random variables $A, B, C$ where $B$ is statistically-independent of $A$ and $C$ (e.g. it is random noise). Consider scalars $0 < \alpha < \beta < 1$ and random variables $X = (...
- 1
0
votes
0
answers
28
views
Entropy of a set of strings based on a sample
Say I have an enormous set of N-character-long strings. Far too many to enumerate or store in memory, but far fewer than the theoretical $26^N$ possible strings. I can draw samples from this set, but ...
- 111
15
votes
2
answers
1k
views
The Bing Tibetan Glitch Emoji Problem
Here's a very interesting mathematical statistics puzzle that I randomly stumbled into while using Bing. This turned out to be deep enough that I spent quite some time thinking how one could solve it! ...
- 415
1
vote
0
answers
59
views
Is a sub Markov chain also a Markov chain?
Let us assume $A \rightarrow B \rightarrow C \rightarrow D$ is a markov chain. Can we also state that $A \rightarrow C \rightarrow D$ is also a Markov chain? It intuitively feels right. Can anyone ...
1
vote
0
answers
16
views
Can a covering argument bound the difference between a batched computation of a function versus the true value?
Suppose we are given a (possibly large) dataset $X \subset \mathcal{X}$ for some complete, separable space $\mathcal{X}$, and a space $\mathcal{Y}$ for which completeness and separability are not ...
- 123
0
votes
0
answers
9
views
LDDP vs info-entropy on binary float representations?
I was just reading about LDDP (Limiting Density of Discrete Points), and I see that it approximates a continuous distribution with an arbitrarily dense discrete distribution. And this reminded me of ...
- 441
0
votes
0
answers
26
views
Query regarding MI and NMI in the context of image fusion
I am exploring Mutual Information (MI) and Normalized Mutual Information (NMI) in the context of image fusion. While reviewing various sources, it's often mentioned that Mutual Information's value ...
- 1
0
votes
0
answers
86
views
Conditions for existence of KL divergence and unique minimum
Consider two probability density function g(y) and f(y: $\theta$), $\theta \in \Theta$.
The KL divergence of f and g is defined by
$$
D_{KL}(g|f) := \int \log \frac{g(y)}{f(y: \theta)} \, dy = \...
2
votes
1
answer
155
views
Conditional mutual information $I(X;Y|Z)$
If there is a function $Y=f(Z)$, is the following mutual information equal to zero?
\begin{align}
I(X;Y|Z)=0
\end{align}
Intuitively, it is correct. But how can we prove this?
- 227
0
votes
0
answers
24
views
KL divergence as minimum patch size for data differencing?
The Wikipedia article on KL divergence mentions a link with data differencing. Directly quoting Wikipedia (as of 2023/11/01):
Just as absolute entropy serves as theoretical background for data ...
- 161
0
votes
0
answers
12
views
How does source and channel coding work without cancelling out?
Source coding is basically compressing raw message bits. Channel coding is adding redundancy. Both are counter to one another. Yet, in a digital communication block diagram, they are always next to ...
- 1
0
votes
0
answers
68
views
Am I understand this problem correct: learning thory, break point, vc dimension
The problem is:
Consider the “triangle” learning model, where $h : R^2 → \{−1, +1\}$ and $h(x) = +1$ if $x$ lies within an arbitrarily chosen triangle in the plane and $−1$ otherwise. Which is the ...
- 185
1
vote
0
answers
53
views
Markov chain data processing inequality
For a Markov chain $X \rightarrow Y \rightarrow Z$, we have the following data processing inequality: $I(Y;X) \ge I(Z;X)$. Now for the Markov chain, $(W,X) \rightarrow Y \rightarrow Z$, can we prove ...
0
votes
0
answers
31
views
Prior probability distribution when we have a single estimate of the mean and no estimate of the variance
Say we have some real parameter $p$ we'd like to determine experimentally. If we have a single estimate of $p$ but no associated uncertainty, what prior probability distribution(s) can/should we use ...
- 684
0
votes
1
answer
38
views
Correlation / information between $X$ and $Z$ not explained through an intermediate variable $Y$
Let $X, Y, Z$ be r.v's. Suppose $X$ has some influence on $Y$ and $Z$, and $Y$ has some influence on $Z$. Can we quantify the influence $X$ has on $Z$, which is not due to the influence $X \rightarrow ...
- 51
1
vote
1
answer
350
views
How to interpret a mutual information value less than 1
For a mutual information of two continuous variables (X and Y), I interpret a value of M.I. = 1 bit (i.e., 2^1 = 2 distinguishable levels), to mean the following:
If I know any given value of X, I ...
- 729
1
vote
1
answer
54
views
Missing something in the calculation of entropy
Problem summary: How does the calculation of entropy differentiate between the "randomness" of two sequences that comprise the same set of elements?
Following this paper, let's say I have ...
- 23
3
votes
1
answer
46
views
Why $\nabla_{\theta} \log p(y;\theta) = \frac{\nabla_{\theta}p(y;\theta)}{p(y;\theta)}$?
I'm solving a problem for cs 229, problem description in below when i check the answer it mentioned the given equation, but I don't undertand why is that.
I want to know why, anyone give me some sort ...
- 185
0
votes
1
answer
70
views
Is CE(X, Y) equivalent to H(X) + H(Y)?
From my understanding mutual information can be defined in the following ways:
[1]:
$I(X;Y)=H(X)+H(Y)-H(X,Y)$ where $H(X), H(Y)$ are marginal entropies and $H(X,Y)$ is the joint entropy.
[2]:
$I(X;Y)=...
- 3
1
vote
0
answers
86
views
Mutual Information between Gaussians in the limiting, strongly correlated case
In this question it is derived that if X, Y are correlated Gaussian variables, the mutual information between them is given by
$$ I(X;Y) = \log\left(\frac{\det(\Sigma_X)\det(\Sigma_Y)}{\det\Sigma_{XY}}...
0
votes
1
answer
86
views
How is the rate $H(X\mid Y)$ achieved in Slepian-Wolf coding?
Given two (generally correlated) sources $X,Y$, Slepian-Wolf coding is a protocol that shows it's possible to encode them separately, then have $X$ send $Y$ only $n H(X\mid Y)$ bits of information, ...
- 383
2
votes
1
answer
115
views
More correlated information is... more informative?
Suppose I am trying to make inference about a parameter $\mu$. I have a prior
$$
\mu \sim N(0,\sigma^2),
$$
and I observe two correlated signals about $\mu,$ namely $x_1, x_2$ where
$$
\begin{pmatrix}
...
- 21
0
votes
1
answer
66
views
Derivation of cross entropy loss in machine learning
Given a dataset $\mathcal{D} = \{ (x_1, y_1),\cdots, (x_n, y_n)\}$, let's say we want to approximate the conditional probability $p(y|x)$, and we parameterized it as $p_{\theta}(y|x)$. So,for a ...
- 101
2
votes
2
answers
597
views
Entropy of an Image?
In a previous question (Entropy of an image) and in various sources on the web, the Shannon entropy of an image is considered to be the entropy of the frequency distribution of the grayscale values.
...
- 21
0
votes
1
answer
56
views
Maximum change in entropy when conditioning on an event
Let $P_{XYZ}$ be the joint distribution of discrete RVs $X,Y,Z$ where $Z$ is binary-valued. Let $Q_{XY}=P_{XY|Z=0}$, i.e. the distribution of $XY$ conditioned on $Z=0$. Are there lower/upper bounds on ...
- 71
1
vote
0
answers
49
views
Calculating co occurrence probabilities of search queries
Hi guys I want to calculate the pointwise mutual information for related search queries on an e-commerce website.
In order to calculate that I need to fist calculate the co occurrence matrix for the ...
1
vote
1
answer
111
views
Can the average log probability score of a model be used as an approximation of the KL divergence?
I'm reading the Chapter 7 of Statistical Rethinking (2nd), where the author delves into information theory and model selection.
I think I've grasped the concept of what would be the KL Divergence, and ...
- 107