Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

In Stata or SAS, how do I run a stacked regression with entropy balancing?

My objective is to run stacked regression (DiD) with entropy balancing in the setting of multiple timing for treatments. Initially, my data consisted of panel data for firm-years. I then stacked this ...
user avatar
2 votes
1 answer
35 views

Bivariate random variable and transformation

Let $X=(X_1,X_2)$ and $Y=(Y_1,Y_2)$ be non-negative absolutely continuous random vector and if $\phi(X_j)=Y_j$, $j=1,2$, are one-one transformation then $$H[Y;\phi(t_1),\phi(t_2)]=H(X;t_1,t_2)-E[\log ...
Unknown's user avatar
  • 163
0 votes
0 answers
24 views

Entropy of a set of strings based on a sample

Say I have an enormous set of N-character-long strings. Far too many to enumerate or store in memory, but far fewer than the theoretical $26^N$ possible strings. I can draw samples from this set, but ...
hjfreyer's user avatar
  • 111
1 vote
0 answers
25 views

Integral Over functions Differential Entropy

Suppose there is some function: \begin{equation} f(t) = p(x) \end{equation} Where $p(x)$ is a PDF over $x$ at $t$. Some examples would be linear regression with error bounds or a Gaussian Process (...
Snowy Baboon's user avatar
1 vote
0 answers
10 views

Analytical expression of mutual information between a multivariate Gaussian and a binary class variable

The goal is to construct settings that allow analytical computations of the mutual information in the form: $$I(X; Y) = H(X) - H(X | Y)$$ I am wondering if that is ...
ajl123's user avatar
  • 287
2 votes
1 answer
46 views

Why is Shannon entropy score lower for distributions with higher variance?

Hi I'm perplexed because I assume that a distribution with higher variance should have higher entropy scores, however this does not appear to be the case? Here is an example. ...
Alex's user avatar
  • 23
0 votes
0 answers
16 views

Mutual information between a vector of values and a statistic computed on the vector

I understand that the mutual information between two vectors X, Y from two distributions gives a measure of uncertainty of X knowing information about Y. But is it possible to measure mutual ...
cryptomath's user avatar
1 vote
1 answer
158 views

Understanding shannon entropy and computation with scipy.stats.entropy

I am trying to understand the shannon entropy better. By definition, the shannon entropy is calculated as H = -sum(pk * log(pk)). I am using the scipy.stats.entropy formula and I am running the ...
GGChe's user avatar
  • 123
6 votes
1 answer
309 views

Why doesn't estimating Shannon entropy with a histogram converge to its true value?

I'm following the third recipe of this answer to estimate the Shannon entropy of my samples using histograms. My expectation was, increasing the sample size should lead to a better estimation of the ...
arash's user avatar
  • 189
0 votes
0 answers
49 views

Why are entropy values in R package mclust not ranged between 0 and 1? [duplicate]

I'm using Mclust to perform a Latent Profile Analysis. I want to observe the entropy values for different cluster solutions. I have used the mclust addons function EntropyGMM, but I find values above ...
nickydv's user avatar
1 vote
1 answer
94 views

Entropy of $\ell^p$ norm of multivariate Gaussian

If $X$ is n-dimensional standard Gaussian, is there an analytic expression for the differential entropy of the $\ell^p$ norm of $X$? For the case $p=2$, the $\ell^2$ norm is exactly the chi ...
Alex Tan's user avatar
0 votes
0 answers
18 views

An unbiased estimator for distance between two observed categorical distributions?

I have two empirical categorical distributions: P and q with |P| >> |q|. P is the full description of my population, so I treat it as the reference baseline distribution. q, however, is a much ...
Ruggiero Spearman's user avatar
1 vote
2 answers
103 views

Tensorization of entropy: confusion regarding conditional entropy

I'm reading High-Dimensional Statistics by Wainright. In the book, entropy for random variable $Z \geq 0$ is defined as $H(Z) = E[Z \log Z]- E[Z] \log E[Z]$. My understanding is that $H(Z)$ is a ...
Phil's user avatar
  • 616
1 vote
1 answer
42 views

Missing something in the calculation of entropy

Problem summary: How does the calculation of entropy differentiate between the "randomness" of two sequences that comprise the same set of elements? Following this paper, let's say I have ...
cmcrowley's user avatar
3 votes
1 answer
98 views

What concentration $\kappa \in [0, \infty)$ maximizes the entropy of the von Mises-Fisher distribution?

I'd like to prove what concentration parameter $\kappa \in [0, \infty)$ maximizes the (differential) entropy of a von-Mises Fisher Distribution. The differential entropy of of a von Mises-Fisher ...
Rylan Schaeffer's user avatar
1 vote
1 answer
75 views

The Tsallis entropy of generalized Gaussian distribution

I would like to discuss the computation of the Tsallis entropy for the generalized Gaussian distribution. From the paper in the link https://www.sciencedirect.com/science/article/pii/S0167947322000822....
M.cadirci's user avatar
1 vote
1 answer
257 views

How to understand the information entropy calculation in entropy weight method?

I found the input of information entropy calculation in the entropy weight method is not probabilities. I can not understand why it works. In most entropy weight method introductions, the information ...
Alex's user avatar
  • 83
0 votes
1 answer
48 views

Weighted entropy using similarity scores

I have a list (freq) of letters with the following frequencies: ...
isthisthat's user avatar
8 votes
1 answer
95 views

What is the complexity to compute the sample entropy of a time series of length N?

background Some algorithms are linear in operations, so the number of operations to complete is $O \left(N\right)$, while others are quadratic $O\left(N^2\right)$, cubic $O \left(N^3\right)$, or non-...
EngrStudent's user avatar
  • 9,205
0 votes
0 answers
14 views

Comparing two different datasets using null deviance

I have two separate datasets, A and B. I have a score that I am applying in both, and I am testing how well that score predicts a disease (present or absent). One property that I think would be ...
carbocation's user avatar
0 votes
1 answer
63 views

Is CE(X, Y) equivalent to H(X) + H(Y)?

From my understanding mutual information can be defined in the following ways: [1]: $I(X;Y)=H(X)+H(Y)-H(X,Y)$ where $H(X), H(Y)$ are marginal entropies and $H(X,Y)$ is the joint entropy. [2]: $I(X;Y)=...
Rui's user avatar
  • 3
0 votes
1 answer
68 views

How is the rate $H(X\mid Y)$ achieved in Slepian-Wolf coding?

Given two (generally correlated) sources $X,Y$, Slepian-Wolf coding is a protocol that shows it's possible to encode them separately, then have $X$ send $Y$ only $n H(X\mid Y)$ bits of information, ...
glS's user avatar
  • 373
1 vote
0 answers
12 views

How to approximate the entropy of convoluted Normal Inverse Wishart distribution?

I am struggling with the (Monte Carlo) approximation of the entropy of convoluted Normal Inverse Wishart distribution. If anyone knows how to do it, I would like to borrow your wisdom. The following ...
tomsoya's user avatar
  • 11
3 votes
1 answer
169 views

Is there an analytical equation for the conditional entropy of a Gaussian random variable?

I am looking for a way to simulate ground-truth conditional entropy. Say I have $\mathbf{X}$ is a 3-dimensional multivariate Gaussian random variable. I am interested in computing the ground-truth of ...
ajl123's user avatar
  • 287
1 vote
0 answers
39 views

Can Shannon-Wiener index be used for measuring temporal differences? [closed]

I understand the Shannon diversity index being used in spatial analysis when comparing multiple sites of measurement, but can it be used to measure abundances, richness, diversity, etc. on a temporal ...
user390865's user avatar
2 votes
2 answers
220 views

Entropy of an Image?

In a previous question (Entropy of an image) and in various sources on the web, the Shannon entropy of an image is considered to be the entropy of the frequency distribution of the grayscale values. ...
Edoardo's user avatar
  • 21
3 votes
1 answer
107 views

Why do we weight the surprisals by the probabilities when computing the entropy?

Let's say that a random variable $X$ takes the values $(x_1, x_2, \dots, x_n)$ over the sample space $\Omega$. As far as my understanding goes, the entropy of a given variable is meant to give an ...
Mehdi Charife's user avatar
0 votes
0 answers
12 views

Stochastic process with constant information gain

For my research I am looking for a stochastic process that fulfills the following conditions. The process takes values between 0 and 1, $X(t) \in [0,1]$ for all $t \in \mathbb{R}$. The process ends ...
HRSE's user avatar
  • 101
1 vote
0 answers
25 views

Finite bracketing integral implies convergence in probability

In a paper I am reading, the following result is used. If $(\mathcal{X},\mathcal{F}) $ is a measurable space, $\mathcal{G}$ is a class of functions with elements $f : \mathcal{X} \to \mathbb{R}$. We ...
Abm's user avatar
  • 352
0 votes
1 answer
47 views

Maximum change in entropy when conditioning on an event

Let $P_{XYZ}$ be the joint distribution of discrete RVs $X,Y,Z$ where $Z$ is binary-valued. Let $Q_{XY}=P_{XY|Z=0}$, i.e. the distribution of $XY$ conditioned on $Z=0$. Are there lower/upper bounds on ...
helloworld's user avatar
0 votes
0 answers
39 views

Is the Wikipedia Sample Entropy code wrong?

I'm following a tutorial paper on sample entropy, and I implemented the algorithm myself in Rust. To check my work, I used Wikipedia's python code in this article. I have computed an example by hand ...
Ryan Folks's user avatar
1 vote
0 answers
25 views

What does ICA using only one component return?

I understand that with multiple components, the result will be coefficients that lead to maximally independent series. When requesting only one component I'm unclear if it actually does optimization ...
user708873's user avatar
0 votes
0 answers
22 views

What do the authors mean by this, regarding estimation of mutual information

I found the following while reading a paper [1] and got confused: Replacing $k_{ij}$, $k{i.}$, respectively $k_{.j}$, by $k_{i,j}$, $k_{i.}$ and $k_{.j}$ provides us with estimates of entropy and ...
Chika's user avatar
  • 1
0 votes
0 answers
16 views

Generalised speed measure balances between entropy of proposal density and average acceptance

This is in reference to page 3 of https://arxiv.org/abs/1911.01373 In the following line after equation (3): the author mentioned that the generalised speed measure (a measure of speed for which a ...
Physkid's user avatar
  • 251
0 votes
0 answers
49 views

Can probabilistic (e.g., Fisher, Shannon) and non-probabilistic (e.g., Hartley, Kolmogorov) information types be jointly useful?

Suppose you draw a random sample from a probability distribution, with the objective of gaining information about a parameter of that distribution. The inferential usefulness of the probabilistic ...
virtuolie's user avatar
  • 528
0 votes
0 answers
59 views

Differential Entropy of Zero-Mean Gaussian Mixtures

Introduction Consider a univariate circularly symmetric complex Gaussian (CSCG) mixture $Y$ with pdf $$p_Y(y) = \sum_i c_i p_i(y) = \sum_i c_i \frac{\exp(-\lvert y \rvert^2/\sigma_i^2)}{\pi \sigma_i^2}...
SnowzTail's user avatar
1 vote
1 answer
89 views

Is the principle of maximum entropy misleading?

If a distribution belongs to a certain class, then the distribution with the largest entropy in that class is typically referred to as the least-informative distribution. To me, this his highly ...
Mr Saltine's user avatar
1 vote
0 answers
15 views

Capturing increased uncertainty as data changes

I'm experimenting with quantifying uncertainty in data from a Bernoulli distribution by measuring the likelihood the p parameter using the beta distribution. Specifically, I'd like to show how ...
exchez's user avatar
  • 111
2 votes
0 answers
86 views

Entropy Balancing Did Not Create Balanced Means (R)

I am looking at how sex impacts judicial decision making and am trying to balance my dataset where my treatment variable is named "gender.judge" (0 for control group, males, and 1 for ...
Liliana's user avatar
  • 21
1 vote
0 answers
124 views

Softmax Response vs MC Dropout for Uncertainty Estimation [closed]

Some papers I see take the uncertainty estimation of a prediction as simply its softmax/sigmoid output, whereas some papers will use techniques such as MC Dropout and calculate the variance across the ...
user8903610's user avatar
0 votes
0 answers
25 views

Model marginal and joint distributions from a sample of unkown number of categories

To illustrate the problems imagine I'm drawing labelled spheres from a box. I may or may not know the number of spheres in the box (does it make a difference?) If I draw 10 spheres from the box and ...
jcp's user avatar
  • 521
0 votes
0 answers
42 views

Conditional Entropy Formula for a Masking Network

I want to prove the following, for a given image distribution $P(X), \mathcal{X} \in \mathbf{R}^{n \times m}$, we have a masking model, $\phi:X \rightarrow \{0,1\}^{n\times m}$. We also have the ...
rando's user avatar
  • 236
0 votes
1 answer
54 views

Descriptive statistics: a metric of the "lumpiness" of numeric vector

I have three datasets of continuous data. Is there a convenient metric for the "binnedness" of the data? How "lumpy" it is? I'd like a single number to allow me to distinguish ...
Carbonyl's user avatar
0 votes
0 answers
39 views

Measuring entropy of a binary matrix with biased probability

I tried to measure entropy of binary matrix like below using code at : https://github.com/cosmoharrigan/matrix-entropy (I already saw the question : Measuring entropy/ information/ patterns of a 2d ...
wsy's user avatar
  • 1
0 votes
0 answers
78 views

How can the mutual information be equal to minus conditional entropy?

I am reading the following paper: https://arxiv.org/abs/2301.06941 The authors in Eq.(8) have obtained a relation which has the mutual information, $i$, in the exponent of the exponential on the RHS ...
user avatar
0 votes
0 answers
160 views

Using zlib to estimate entropy rate of text data

I'm trying to estimate the entropy rate of a text file (about 2 million characters, including spacing and punctuation). Calculating the entropy rate directly is not an option for me, for lack of ...
MarcoLin8's user avatar
0 votes
0 answers
165 views

Marginal contribution of each observation to posterior moments and (differential) entropy are decreasing in the number of observations?

In the Beta-Binomial and Gaussian settings, indeed, the marginal contribution of each observation to the posterior moments (I could check first and second) and (differential) entropy are decreasing in ...
Ararat's user avatar
  • 93
0 votes
1 answer
59 views

Why the maximum of Rényi entropy is not achieved for the uniform distribution? [closed]

Given a discrete random variable $X$, which takes values in the alphabet $\mathcal {X}$ and is distributed according to $p:{\mathcal {X}}\to [0,1]$ the Shannon entropy is defined as: $$\mathrm {H} (X):...
Mark's user avatar
  • 307
1 vote
0 answers
126 views

How to calculate a path entropy in the context of information theory?

Consider a path where the system goes from states $i$ to $j$ and then from $j$ to $k$. Given that the probability of finding the system in state $i$ is $p_i$ and the transition probabilities are $p_{...
Jack's user avatar
  • 71
1 vote
0 answers
78 views

Understanding properties of conditional entropy

I was trying to understand concept of cross entropy. Wikipedia says: $H(Y|X)\leq H(Y)$ Although the specific conditional entropy $H(X|Y=y)$ can be either less or greater than $H(X)$ for a given ...
Mahesha999's user avatar

1
2 3 4 5
14