# Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

695 questions
Filter by
Sorted by
Tagged with
20 views

### differential entropy for comparison distributions

I want to use differential entropy to compare the outcome of Bayesian updating (multidimensional probability distributions) for different datasets. My parameters are different physical parameters i.e. ...
• 41
1 vote
24 views

### How should I go about completely decorrelating a digital signal?

So I'm working on real time signal compression, and I need to come up with the best convolution to minimize the entropy of incoming data (which I will then compress), which I understand is achieved by ...
• 11
18 views

### Interpretation of time series spectral entropy values wrt forecastability by a general neural network

I recently started using spectral entropy to analyze time series (already windowed). I'm having difficulty for interpreting the results, the entropy of the last 25% of a series is 0.19, and the ...
• 51
41 views

### Shannon source coding theorem and differential entropy

Loosely speaking, Shannon's source encoding theorem says that there is an encoder with rate at least $H(x)$ such that $n$ repetitions of the source can be mapped to at least $nH(X)$ bits of binary ...
• 121
1 vote
60 views

### Chain rule conditional entropy

A textbook I am reading states that$$H(X,Y)=H(X)+H(Y|X)$$where $H(X,Y)$ is the joint entropy of random variables $X,Y$, $H(X)$ the entropy of $X$, and $H(Y|X)$ is conditional entropy. It then states ...
• 125
1 vote
44 views

• 3
83 views

### How is the rate $H(X\mid Y)$ achieved in Slepian-Wolf coding?

Given two (generally correlated) sources $X,Y$, Slepian-Wolf coding is a protocol that shows it's possible to encode them separately, then have $X$ send $Y$ only $n H(X\mid Y)$ bits of information, ...
• 383
1 vote
14 views

### How to approximate the entropy of convoluted Normal Inverse Wishart distribution?

I am struggling with the (Monte Carlo) approximation of the entropy of convoluted Normal Inverse Wishart distribution. If anyone knows how to do it, I would like to borrow your wisdom. The following ...
• 11
316 views

### Is there an analytical equation for the conditional entropy of a Gaussian random variable?

I am looking for a way to simulate ground-truth conditional entropy. Say I have $\mathbf{X}$ is a 3-dimensional multivariate Gaussian random variable. I am interested in computing the ground-truth of ...
• 307
1 vote
48 views

### Can Shannon-Wiener index be used for measuring temporal differences? [closed]

I understand the Shannon diversity index being used in spatial analysis when comparing multiple sites of measurement, but can it be used to measure abundances, richness, diversity, etc. on a temporal ...
496 views

### Entropy of an Image?

In a previous question (Entropy of an image) and in various sources on the web, the Shannon entropy of an image is considered to be the entropy of the frequency distribution of the grayscale values. ...
• 21
137 views

### Why do we weight the surprisals by the probabilities when computing the entropy?

Let's say that a random variable $X$ takes the values $(x_1, x_2, \dots, x_n)$ over the sample space $\Omega$. As far as my understanding goes, the entropy of a given variable is meant to give an ...
29 views

### Finite bracketing integral implies convergence in probability

In a paper I am reading, the following result is used. If $(\mathcal{X},\mathcal{F})$ is a measurable space, $\mathcal{G}$ is a class of functions with elements $f : \mathcal{X} \to \mathbb{R}$. We ...
• 382
53 views

### Maximum change in entropy when conditioning on an event

Let $P_{XYZ}$ be the joint distribution of discrete RVs $X,Y,Z$ where $Z$ is binary-valued. Let $Q_{XY}=P_{XY|Z=0}$, i.e. the distribution of $XY$ conditioned on $Z=0$. Are there lower/upper bounds on ...
1 vote
32 views

### What does ICA using only one component return?

I understand that with multiple components, the result will be coefficients that lead to maximally independent series. When requesting only one component I'm unclear if it actually does optimization ...
23 views

### What do the authors mean by this, regarding estimation of mutual information

I found the following while reading a paper [1] and got confused: Replacing $k_{ij}$, $k{i.}$, respectively $k_{.j}$, by $k_{i,j}$, $k_{i.}$ and $k_{.j}$ provides us with estimates of entropy and ...
1 vote
96 views

### Is the principle of maximum entropy misleading?

If a distribution belongs to a certain class, then the distribution with the largest entropy in that class is typically referred to as the least-informative distribution. To me, this his highly ...
1 vote
16 views

### Capturing increased uncertainty as data changes

I'm experimenting with quantifying uncertainty in data from a Bernoulli distribution by measuring the likelihood the p parameter using the beta distribution. Specifically, I'd like to show how ...
• 111
117 views

### Entropy Balancing Did Not Create Balanced Means (R)

I am looking at how sex impacts judicial decision making and am trying to balance my dataset where my treatment variable is named "gender.judge" (0 for control group, males, and 1 for ...
• 21
1 vote
168 views

### Softmax Response vs MC Dropout for Uncertainty Estimation [closed]

Some papers I see take the uncertainty estimation of a prediction as simply its softmax/sigmoid output, whereas some papers will use techniques such as MC Dropout and calculate the variance across the ...
25 views

### Model marginal and joint distributions from a sample of unkown number of categories

To illustrate the problems imagine I'm drawing labelled spheres from a box. I may or may not know the number of spheres in the box (does it make a difference?) If I draw 10 spheres from the box and ...
• 521