Skip to main content

Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

differential entropy for comparison distributions

I want to use differential entropy to compare the outcome of Bayesian updating (multidimensional probability distributions) for different datasets. My parameters are different physical parameters i.e. ...
Sobol's user avatar
  • 41
1 vote
0 answers
24 views

How should I go about completely decorrelating a digital signal?

So I'm working on real time signal compression, and I need to come up with the best convolution to minimize the entropy of incoming data (which I will then compress), which I understand is achieved by ...
2 False's user avatar
  • 11
0 votes
0 answers
18 views

Interpretation of time series spectral entropy values wrt forecastability by a general neural network

I recently started using spectral entropy to analyze time series (already windowed). I'm having difficulty for interpreting the results, the entropy of the last 25% of a series is 0.19, and the ...
Marco's user avatar
  • 51
2 votes
0 answers
41 views

Shannon source coding theorem and differential entropy

Loosely speaking, Shannon's source encoding theorem says that there is an encoder with rate at least $H(x)$ such that $n$ repetitions of the source can be mapped to at least $nH(X)$ bits of binary ...
nervxxx's user avatar
  • 121
1 vote
1 answer
60 views

Chain rule conditional entropy

A textbook I am reading states that$$H(X,Y)=H(X)+H(Y|X)$$where $H(X,Y)$ is the joint entropy of random variables $X,Y$, $H(X)$ the entropy of $X$, and $H(Y|X)$ is conditional entropy. It then states ...
user124910's user avatar
1 vote
1 answer
44 views

Supposed to be a simple question about entropy

Let say there is an urn which contains balls of different color. It is a well known formula to calculate entropy of balls in the urn: $H = - \sum P_i\cdot\log(P_i)$ where $P_i = \frac{M_i}{N}$, where $...
irriss's user avatar
  • 121
0 votes
1 answer
94 views

Check if my time series is forecastable using Shannon entropy

According to this answer: https://datascience.stackexchange.com/a/95232/141037, is possible to verify the forecastability of a time series using the Shannon entropy, the lower the Shannon entropy ...
Marco's user avatar
  • 51
2 votes
0 answers
22 views

How to quantify "clumpiness" of a time series of physical activity?

Let's assume you are measuring the physical activity levels over a day of some people, using accelerometry for example. The goal is to quantify the "clumpiness" of the activity patterns. It ...
COOLSerdash's user avatar
1 vote
1 answer
44 views

Minimum entropy decomposition of probability distributions

Say you want to decompose a probability distribution (a PDF) into a mixture of distributions in such a way as to minimize the mean entropy of the component distributions. I have an idea that this is ...
zonofzin's user avatar
0 votes
0 answers
19 views

How to use and understand entropy for pattern detection?

I have two images from erp_data and noerp_data matricies. In erp_data we can see a pattern (sigmoid), in no_erp we see no pattern. ERP is event-related potential, if you are curious. My goal is to ...
Vladimir Mikheev's user avatar
1 vote
0 answers
16 views

Effect on entropy when we scale Bernoulli plus Gaussian

Question: Given $X\sim\text{Bernoulli}(\alpha)$, $Y\sim\mathcal{N}(0,1)$, and non-random positive constants $C,\epsilon>0$. Let $H(\cdot)$ be the differential entropy. Is it true that $$ H((C+\...
Resu's user avatar
  • 229
0 votes
0 answers
75 views

In Stata or SAS, how do I run a stacked regression with entropy balancing?

My objective is to run stacked regression (DiD) with entropy balancing in the setting of multiple timing for treatments. Initially, my data consisted of panel data for firm-years. I then stacked this ...
user avatar
2 votes
1 answer
42 views

Bivariate random variable and transformation

Let $X=(X_1,X_2)$ and $Y=(Y_1,Y_2)$ be non-negative absolutely continuous random vector and if $\phi(X_j)=Y_j$, $j=1,2$, are one-one transformation then $$H[Y;\phi(t_1),\phi(t_2)]=H(X;t_1,t_2)-E[\log ...
Unknown's user avatar
  • 173
0 votes
0 answers
28 views

Entropy of a set of strings based on a sample

Say I have an enormous set of N-character-long strings. Far too many to enumerate or store in memory, but far fewer than the theoretical $26^N$ possible strings. I can draw samples from this set, but ...
hjfreyer's user avatar
  • 111
1 vote
0 answers
27 views

Integral Over functions Differential Entropy

Suppose there is some function: \begin{equation} f(t) = p(x) \end{equation} Where $p(x)$ is a PDF over $x$ at $t$. Some examples would be linear regression with error bounds or a Gaussian Process (...
Snowy Baboon's user avatar
1 vote
0 answers
13 views

Analytical expression of mutual information between a multivariate Gaussian and a binary class variable

The goal is to construct settings that allow analytical computations of the mutual information in the form: $$I(X; Y) = H(X) - H(X | Y)$$ I am wondering if that is ...
ajl123's user avatar
  • 307
0 votes
0 answers
6 views

LDDP vs info-entropy on binary float representations?

I was just reading about LDDP (Limiting Density of Discrete Points), and I see that it approximates a continuous distribution with an arbitrarily dense discrete distribution. And this reminded me of ...
profPlum's user avatar
  • 411
2 votes
1 answer
57 views

Why is Shannon entropy score lower for distributions with higher variance?

Hi I'm perplexed because I assume that a distribution with higher variance should have higher entropy scores, however this does not appear to be the case? Here is an example. ...
Alex's user avatar
  • 23
0 votes
0 answers
24 views

Mutual information between a vector of values and a statistic computed on the vector

I understand that the mutual information between two vectors X, Y from two distributions gives a measure of uncertainty of X knowing information about Y. But is it possible to measure mutual ...
cryptomath's user avatar
1 vote
1 answer
308 views

Understanding shannon entropy and computation with scipy.stats.entropy

I am trying to understand the shannon entropy better. By definition, the shannon entropy is calculated as H = -sum(pk * log(pk)). I am using the scipy.stats.entropy formula and I am running the ...
GGChe's user avatar
  • 165
6 votes
1 answer
369 views

Why doesn't estimating Shannon entropy with a histogram converge to its true value?

I'm following the third recipe of this answer to estimate the Shannon entropy of my samples using histograms. My expectation was, increasing the sample size should lead to a better estimation of the ...
arash's user avatar
  • 309
0 votes
0 answers
49 views

Why are entropy values in R package mclust not ranged between 0 and 1? [duplicate]

I'm using Mclust to perform a Latent Profile Analysis. I want to observe the entropy values for different cluster solutions. I have used the mclust addons function EntropyGMM, but I find values above ...
nickydv's user avatar
1 vote
1 answer
106 views

Entropy of $\ell^p$ norm of multivariate Gaussian

If $X$ is n-dimensional standard Gaussian, is there an analytic expression for the differential entropy of the $\ell^p$ norm of $X$? For the case $p=2$, the $\ell^2$ norm is exactly the chi ...
Alex Tan's user avatar
0 votes
0 answers
29 views

An unbiased estimator for distance between two observed categorical distributions?

I have two empirical categorical distributions: P and q with |P| >> |q|. P is the full description of my population, so I treat it as the reference baseline distribution. q, however, is a much ...
Ruggiero Spearman's user avatar
1 vote
2 answers
140 views

Tensorization of entropy: confusion regarding conditional entropy

I'm reading High-Dimensional Statistics by Wainright. In the book, entropy for random variable $Z \geq 0$ is defined as $H(Z) = E[Z \log Z]- E[Z] \log E[Z]$. My understanding is that $H(Z)$ is a ...
Phil's user avatar
  • 636
1 vote
1 answer
53 views

Missing something in the calculation of entropy

Problem summary: How does the calculation of entropy differentiate between the "randomness" of two sequences that comprise the same set of elements? Following this paper, let's say I have ...
cmc's user avatar
  • 23
3 votes
1 answer
116 views

What concentration $\kappa \in [0, \infty)$ maximizes the entropy of the von Mises-Fisher distribution?

I'd like to prove what concentration parameter $\kappa \in [0, \infty)$ maximizes the (differential) entropy of a von-Mises Fisher Distribution. The differential entropy of of a von Mises-Fisher ...
Rylan Schaeffer's user avatar
1 vote
1 answer
87 views

The Tsallis entropy of generalized Gaussian distribution

I would like to discuss the computation of the Tsallis entropy for the generalized Gaussian distribution. From the paper in the link https://www.sciencedirect.com/science/article/pii/S0167947322000822....
M.cadirci's user avatar
2 votes
1 answer
455 views

How to understand the information entropy calculation in entropy weight method?

I found the input of information entropy calculation in the entropy weight method is not probabilities. I can not understand why it works. In most entropy weight method introductions, the information ...
Alex's user avatar
  • 93
0 votes
1 answer
63 views

Weighted entropy using similarity scores

I have a list (freq) of letters with the following frequencies: ...
isthisthat's user avatar
8 votes
1 answer
112 views

What is the complexity to compute the sample entropy of a time series of length N?

background Some algorithms are linear in operations, so the number of operations to complete is $O \left(N\right)$, while others are quadratic $O\left(N^2\right)$, cubic $O \left(N^3\right)$, or non-...
EngrStudent's user avatar
  • 9,580
0 votes
0 answers
15 views

Comparing two different datasets using null deviance

I have two separate datasets, A and B. I have a score that I am applying in both, and I am testing how well that score predicts a disease (present or absent). One property that I think would be ...
carbocation's user avatar
0 votes
1 answer
66 views

Is CE(X, Y) equivalent to H(X) + H(Y)?

From my understanding mutual information can be defined in the following ways: [1]: $I(X;Y)=H(X)+H(Y)-H(X,Y)$ where $H(X), H(Y)$ are marginal entropies and $H(X,Y)$ is the joint entropy. [2]: $I(X;Y)=...
Rui's user avatar
  • 3
0 votes
1 answer
83 views

How is the rate $H(X\mid Y)$ achieved in Slepian-Wolf coding?

Given two (generally correlated) sources $X,Y$, Slepian-Wolf coding is a protocol that shows it's possible to encode them separately, then have $X$ send $Y$ only $n H(X\mid Y)$ bits of information, ...
glS's user avatar
  • 383
1 vote
0 answers
14 views

How to approximate the entropy of convoluted Normal Inverse Wishart distribution?

I am struggling with the (Monte Carlo) approximation of the entropy of convoluted Normal Inverse Wishart distribution. If anyone knows how to do it, I would like to borrow your wisdom. The following ...
tomsoya's user avatar
  • 11
3 votes
1 answer
316 views

Is there an analytical equation for the conditional entropy of a Gaussian random variable?

I am looking for a way to simulate ground-truth conditional entropy. Say I have $\mathbf{X}$ is a 3-dimensional multivariate Gaussian random variable. I am interested in computing the ground-truth of ...
ajl123's user avatar
  • 307
1 vote
0 answers
48 views

Can Shannon-Wiener index be used for measuring temporal differences? [closed]

I understand the Shannon diversity index being used in spatial analysis when comparing multiple sites of measurement, but can it be used to measure abundances, richness, diversity, etc. on a temporal ...
user390865's user avatar
2 votes
2 answers
496 views

Entropy of an Image?

In a previous question (Entropy of an image) and in various sources on the web, the Shannon entropy of an image is considered to be the entropy of the frequency distribution of the grayscale values. ...
Edoardo's user avatar
  • 21
3 votes
1 answer
137 views

Why do we weight the surprisals by the probabilities when computing the entropy?

Let's say that a random variable $X$ takes the values $(x_1, x_2, \dots, x_n)$ over the sample space $\Omega$. As far as my understanding goes, the entropy of a given variable is meant to give an ...
Mehdi Charife's user avatar
2 votes
0 answers
29 views

Finite bracketing integral implies convergence in probability

In a paper I am reading, the following result is used. If $(\mathcal{X},\mathcal{F}) $ is a measurable space, $\mathcal{G}$ is a class of functions with elements $f : \mathcal{X} \to \mathbb{R}$. We ...
Abm's user avatar
  • 382
0 votes
1 answer
53 views

Maximum change in entropy when conditioning on an event

Let $P_{XYZ}$ be the joint distribution of discrete RVs $X,Y,Z$ where $Z$ is binary-valued. Let $Q_{XY}=P_{XY|Z=0}$, i.e. the distribution of $XY$ conditioned on $Z=0$. Are there lower/upper bounds on ...
helloworld's user avatar
1 vote
0 answers
32 views

What does ICA using only one component return?

I understand that with multiple components, the result will be coefficients that lead to maximally independent series. When requesting only one component I'm unclear if it actually does optimization ...
user708873's user avatar
0 votes
0 answers
23 views

What do the authors mean by this, regarding estimation of mutual information

I found the following while reading a paper [1] and got confused: Replacing $k_{ij}$, $k{i.}$, respectively $k_{.j}$, by $k_{i,j}$, $k_{i.}$ and $k_{.j}$ provides us with estimates of entropy and ...
Chika's user avatar
  • 1
1 vote
1 answer
96 views

Is the principle of maximum entropy misleading?

If a distribution belongs to a certain class, then the distribution with the largest entropy in that class is typically referred to as the least-informative distribution. To me, this his highly ...
Mr Saltine's user avatar
1 vote
0 answers
16 views

Capturing increased uncertainty as data changes

I'm experimenting with quantifying uncertainty in data from a Bernoulli distribution by measuring the likelihood the p parameter using the beta distribution. Specifically, I'd like to show how ...
exchez's user avatar
  • 111
2 votes
0 answers
117 views

Entropy Balancing Did Not Create Balanced Means (R)

I am looking at how sex impacts judicial decision making and am trying to balance my dataset where my treatment variable is named "gender.judge" (0 for control group, males, and 1 for ...
Liliana's user avatar
  • 21
1 vote
0 answers
168 views

Softmax Response vs MC Dropout for Uncertainty Estimation [closed]

Some papers I see take the uncertainty estimation of a prediction as simply its softmax/sigmoid output, whereas some papers will use techniques such as MC Dropout and calculate the variance across the ...
user8903610's user avatar
0 votes
0 answers
25 views

Model marginal and joint distributions from a sample of unkown number of categories

To illustrate the problems imagine I'm drawing labelled spheres from a box. I may or may not know the number of spheres in the box (does it make a difference?) If I draw 10 spheres from the box and ...
jcp's user avatar
  • 521
0 votes
1 answer
65 views

Descriptive statistics: a metric of the "lumpiness" of numeric vector

I have three datasets of continuous data. Is there a convenient metric for the "binnedness" of the data? How "lumpy" it is? I'd like a single number to allow me to distinguish ...
Carbonyl's user avatar
0 votes
0 answers
222 views

Using zlib to estimate entropy rate of text data

I'm trying to estimate the entropy rate of a text file (about 2 million characters, including spacing and punctuation). Calculating the entropy rate directly is not an option for me, for lack of ...
MarcoLin8's user avatar

1
2 3 4 5
14