# Questions tagged [information-theory]

A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in predicting a random variable.

674 questions
Filter by
Sorted by
Tagged with
12 views

### Mutual Information of nonadjacent nodes in Bayesian Network

How do you compute the mutual information of two non-adjacent nodes in a Bayesian network? In this case, what would $I(D;A)$ be? Would I need to take the conditional probabilities of all intemediate ...
28 views

### differential entropy for comparison distributions

I want to use differential entropy to compare the outcome of Bayesian updating (multidimensional probability distributions) for different datasets. My parameters are different physical parameters i.e. ...
• 61
105 views

### Is feature-extraction and dimensionality-reduction a kind of compression?

I'm struggling to understand what these terms have in common: Feature extraction Feature selection Compression Dimensionality reduction Relatedly, the information / entropy in our data should always ...
• 23
1 vote
27 views

### Fourier transform in information transfer in biological neural network

Principles of Neural Design by Peter Sterling and Simon Laughlin describes a usage of information theory in calculating the rate of information transfer in the brain. ...when successive signal states ...
4 views

### Sample Complexity of BHT with varying degrees of (large) compression

In Communication-constrained hypothesis testing: Optimality, robustness, and reverse data processing inequalities, the following (up to some mild editing to highlight my question) is established. ...
19 views

### Interpretation of time series spectral entropy values wrt forecastability by a general neural network

I recently started using spectral entropy to analyze time series (already windowed). I'm having difficulty for interpreting the results, the entropy of the last 25% of a series is 0.19, and the ...
• 51
42 views

### Shannon source coding theorem and differential entropy

Loosely speaking, Shannon's source encoding theorem says that there is an encoder with rate at least $H(x)$ such that $n$ repetitions of the source can be mapped to at least $nH(X)$ bits of binary ...
• 121
66 views

### Chain rule conditional entropy

A textbook I am reading states that$$H(X,Y)=H(X)+H(Y|X)$$where $H(X,Y)$ is the joint entropy of random variables $X,Y$, $H(X)$ the entropy of $X$, and $H(Y|X)$ is conditional entropy. It then states ...
• 135
21 views

### How do you choose to put a distribution on the right or left of KL divergence? [duplicate]

I always thought of KL divergence as a distance metric between distributions, much like Earth-Movers distance. But I can no longer ignore the asymmetry. A real distance metric is symmetric. How should ...
• 431
1 vote
28 views

### Mutual Information decay

Consider $m$ channels indexed by $i$ with $1 \leq i \leq m$. The input alphabets are from the same finite set $\mathcal{X}$. Let $\pi$ denote a probability distribution on $\mathcal{X}$. Define the ...
286 views

### Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation

TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this. This is how I understand Minimum ...
• 187
43 views

### Is there any work linking information channel theory to statistical inference?

I wonder what is the theoretical limit of a statistical inference problem. For example we have a model with many parameters, and we can sample many data points from the model. This can be viewed as a ...
1 vote
29 views

### Choosing number of lag AND model form for Augmented Dickey-Fuller test

Before realising an Augmented Dickey-Fuller (ADF) test, one has to answer 2 questions, how many lags p to include in the model, AND which model to choose among the following: No constant, no trend ...
• 11
1 vote
45 views

### Minimum entropy decomposition of probability distributions

Say you want to decompose a probability distribution (a PDF) into a mixture of distributions in such a way as to minimize the mean entropy of the component distributions. I have an idea that this is ...
• 11
1 vote
16 views

95 views

### Conditional mutual information $I(X;Y|Z)$

If there is a function $Y=f(Z)$, is the following mutual information equal to zero? \begin{align} I(X;Y|Z)=0 \end{align} Intuitively, it is correct. But how can we prove this?
• 227
21 views

### KL divergence as minimum patch size for data differencing?

The Wikipedia article on KL divergence mentions a link with data differencing. Directly quoting Wikipedia (as of 2023/11/01): Just as absolute entropy serves as theoretical background for data ...
• 151
12 views

### How does source and channel coding work without cancelling out?

Source coding is basically compressing raw message bits. Channel coding is adding redundancy. Both are counter to one another. Yet, in a digital communication block diagram, they are always next to ...
63 views

### Am I understand this problem correct: learning thory, break point, vc dimension

The problem is: Consider the “triangle” learning model, where $h : R^2 → \{−1, +1\}$ and $h(x) = +1$ if $x$ lies within an arbitrarily chosen triangle in the plane and $−1$ otherwise. Which is the ...
• 185
1 vote
52 views

### Markov chain data processing inequality

For a Markov chain $X \rightarrow Y \rightarrow Z$, we have the following data processing inequality: $I(Y;X) \ge I(Z;X)$. Now for the Markov chain, $(W,X) \rightarrow Y \rightarrow Z$, can we prove ...
30 views

### Prior probability distribution when we have a single estimate of the mean and no estimate of the variance

Say we have some real parameter $p$ we'd like to determine experimentally. If we have a single estimate of $p$ but no associated uncertainty, what prior probability distribution(s) can/should we use ...
• 684
38 views

• 3
1 vote
70 views

• 21
61 views

### Derivation of cross entropy loss in machine learning

Given a dataset $\mathcal{D} = \{ (x_1, y_1),\cdots, (x_n, y_n)\}$, let's say we want to approximate the conditional probability $p(y|x)$, and we parameterized it as $p_{\theta}(y|x)$. So,for a ...
• 101
513 views

### Entropy of an Image?

In a previous question (Entropy of an image) and in various sources on the web, the Shannon entropy of an image is considered to be the entropy of the frequency distribution of the grayscale values. ...
• 21
53 views

### Maximum change in entropy when conditioning on an event

Let $P_{XYZ}$ be the joint distribution of discrete RVs $X,Y,Z$ where $Z$ is binary-valued. Let $Q_{XY}=P_{XY|Z=0}$, i.e. the distribution of $XY$ conditioned on $Z=0$. Are there lower/upper bounds on ...
1 vote
45 views

### Calculating co occurrence probabilities of search queries

Hi guys I want to calculate the pointwise mutual information for related search queries on an e-commerce website. In order to calculate that I need to fist calculate the co occurrence matrix for the ...
1 vote
94 views

### Can the average log probability score of a model be used as an approximation of the KL divergence?

I'm reading the Chapter 7 of Statistical Rethinking (2nd), where the author delves into information theory and model selection. I think I've grasped the concept of what would be the KL Divergence, and ...
• 107
1 vote
33 views

### What does ICA using only one component return?

I understand that with multiple components, the result will be coefficients that lead to maximally independent series. When requesting only one component I'm unclear if it actually does optimization ...
1 vote
168 views

### Mutual information I(X, Y) >= I(f(X), f(Y)) for deterministic f?

I have the intuition that applying a deterministic function to a pair random variables cannot increase their mutual information, because the function can only decrease each of their entropies. I would ...
• 521
1 vote
19 views

### How do I numerically compute $I(X;CX+Y)$?

Given that $X\sim\text{Bernoulli}(\nu)$ for some $\nu\in(0,1)$, and $Y\sim N(0,1)$ are independent random variables. I want to compute the mutual information $I(X;CX+Y)$, where $C$ is some known non-...
• 229
1 vote
### What is the sum of $H(CX+Y)$ when $X$ and $Y$ are independent?
Given that $X\sim\text{Bernoulli}(\nu)$, for some $\nu\in(0,1)$, and $Y\sim N(0,1)$ are independent random variables. What is the entropy $H(CX+Y)$, where $C$ is some fixed constant? I am a bit ...