Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Determining history length in Transfer Entropy

Supposing I have a time series observation corresponding to 5 sensors, and I wish to find the causation(Transfer entropy) between all pairs of nodes, how do I go about choosing the history length ...
2
votes
1answer
30 views

Measurement of regularity of events

I am dealing with events generated by different processes. My data includes time stamps when the events occur, so I am trying to differentiate the processes in regular or irregular categories or ...
0
votes
0answers
4 views

How to calculate the uncertainty of a variable explained by other variables

I'm working on implementing a method from a paper for solving the FRn-k problem for a specific type of biological data. [FRn−k]: Given an initial set of n features, find the subset with k<n ...
2
votes
1answer
35 views

Differential Entropy with negative support

Differential Entropy Estimation Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. ...
3
votes
1answer
124 views

Is there a name for this in statisitics? [closed]

Edit: Ignore this Most of this doesn't make sense and is beyond edits but its too late to delete. If I make too many edits I could be kicked out of this sub like I was in math stack exchange. I don't ...
3
votes
1answer
39 views

Mutual information is not zero for independent variables and negative for weakly dependency

To the best of my knowledge, mutual information (MI) is zero if and only if the variables are independent. I have simulated copula data and computed the ...
0
votes
0answers
13 views

Negative Tsallis entropy?

I am building (many years later !) on this SO question: https://stackoverflow.com/questions/22461241/tsallis-entropy-for-continuous-variable-in-r I should make clear that I'm a stats newbie, exploring ...
4
votes
1answer
44 views

How are entropy and Gini Impurity related?

I know the differences between entropy and Gini impurity and why we use Gini in order to construct trees. But I would like to find some relation between those two measures. It leads me to one ...
0
votes
0answers
26 views

Lower bound on conditional entropy when conditioning on event

In Lemma 3.8 of this paper, the following is shown for jointly distributed random variables $Z$, $W$, where $Z$ takes values in $\{0,1\}^n$, and some event $\mathcal{E}$: $$ H[ Z \mid W ] \ge n - d\ \...
0
votes
0answers
25 views

Gradient with respect to a probability distribution

I'm trying to understand the worked propositions on this paper (reinforcement learning): https://arxiv.org/abs/2007.02832 The authors formulate this objective: $\hat{g}^*=\arg\max_{\hat{g}\in B}\...
2
votes
1answer
78 views

Base of the logarithm used for Shannon Entropy when considering a vocabulary

If we want to calculate the shannon entropy of a text, considering that the elementary symbols aren't characters but words, will we still use a base-2 logarithm ? Or will we use a base-n logarithm ...
1
vote
0answers
34 views

KL divergence from a family of distributions

The KL divergence between two probability measures $q,p$ is $$KL(q(x)||p(x)) = - \int_{\mathcal{X}} q(x) \log\frac{p(x)}{q(x)} dx.$$ What's the KL divergence between a measure and a set of measures? ...
1
vote
1answer
51 views

Does comparing Shannon Entropy between categorical variables makes any sense?

I want to present a report that highlights the variables with lowest information, to emphasize that some action must be taken by the department that controls our data source. I've applied Shannon ...
0
votes
1answer
33 views

Entropy of gaussian mixture

Does the entropy of a gaussian mixture depend on its means? It is not the case for a single Gaussian and when the components of the mixture are far spread out, we can approximate the entropy by a ...
0
votes
0answers
34 views

What is extropy?

While searching for the answer to a statistics problem, I came across the term extropy, which was - and to a large extent still is - unfamiliar to me. A quick google search revealed that it is the &...
0
votes
0answers
4 views

Choosing appropriate values for $m$ and $r$ in approximate entropy

Is there a standard way to determine optimal values for $m$ and $r$ in approximate entropy, maybe based on some quantitative criteria or statistics of the time series? Wikipedia defines The value of $...
0
votes
0answers
41 views

Example of mutual information calculated with R packages in the case of continuous normal variables

Nassim Taleb mentions mutual information as an alternative to correlation, given the tendency of correlation to be very different from zero even in draws from a normal distribution in a simulation. I ...
0
votes
1answer
63 views

How can we determine Conditional Mutual Information based on multiple conditions

Let X,Y,Z be jointly distributed, the conditional mutual information is defined as :- Similarly if we have 4 (or more variables) say X,Y,Z,W then how can we determine the conditional mutual ...
0
votes
0answers
19 views

Scale independent entropy?

Let $i \mapsto f(i)$ denote a probability distribution on a discrete space $I_n=\{1,\dots,n\}$. It is common to use Shannon entropy $H(n) = -\sum f(i)\log f(i)$ as a measure of uncertainty, where I ...
2
votes
1answer
24 views

Can anyone suggest minimum and maximum values that the relative entropy/KL divergence can range when computed from two time-series sequences?

Can anyone suggest minimum and maximum values that the relative entropy/KL divergence can range when computed from two time-series sequences? I have calculated the relative entropy for two time-series ...
0
votes
2answers
64 views

Is it true that entropy estimation is meaningless if samples are not i.i.d.?

Let $X$ be a discrete-support stochastic variable. Information entropy is a number defined as $$H(X)=-\sum_{n} p\left(x_{n}\right) \log p\left(x_{n}\right) \geq 0$$ Let $\hat{H}$ be an estimation of $...
2
votes
0answers
28 views

Why does Kullback–Leibler divergence measure information loss when approximating a probability distribution?

I've encountered a sentence: In information theory, Kullback–Leibler divergence is regarded as a measure of the information lost when probability distribution Q is used to approximate a true ...
0
votes
0answers
25 views

Showing that conditional entropies $H(X \mid Y)$ and $H(Y \mid X)$ are equal for delta distributions

I need some help with the following problem. What does it mean for the joint probability distribution of two random variables $x$ and $y$, $p(x,y)$ , to be equal to $\delta (x,y)$ (Kronecker delta)?...
1
vote
0answers
15 views

How am I getting this entropy calculation wrong?

I am following the entropy calculation in 1a here. Shouldn't the solution for H(Edible | Odor = 1 or 3) be like this? $H = -\frac{3}{6}\log_2{\frac{3}{6}} = \frac{1}{2}$ My reasoning is that we ...
2
votes
1answer
29 views

How can I show that the entropy of a function of random variables cannot be greater than their joint entropy?

Given the discrete random variables $X,Y,$ and $Z=f(X,Y)$, where $f$ is some function, how can I show that: $$ H(Z) \leq H(X,Y) $$ With equality if the function $f$ is invertible?
2
votes
1answer
57 views

Controlling the entropy of a distribution

What is the best way to control the entropy of a categorical probability distribution? I have a categorical probability distribution $p_1, p_2, \dots, p_K$, for $K$ small-ish ($< 1000$). Assume ...
0
votes
0answers
23 views

How do we get from entropy to KL divergence in this paper?

I'm reading through Regularizing Neural Networks By Penalizing Confident Output Distributions and I'm stuck on the equation in section 3.2. It's not clear to me at all that the self-entropy of the ...
1
vote
1answer
116 views

Intuition on Wasserstein Distance

I've been trying to familiarize myself with the Wasserstein distance and saw this answer on StackExchange by @antike that at first made a lot of sense, but then it didn't (to me, of course). In the ...
3
votes
0answers
221 views

Geometric distribution and entropy

According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{ \mu} $ ...
1
vote
0answers
32 views

Generate data that matches a frequency distribution while preserving the original spatial structure

I am dealing with a 3D array containing values representing the "importance" of each voxel. For my analysis, I would like to synthesize n new arrays from my original array to have a ...
0
votes
1answer
70 views

Difference between mutual and conditional information

I have studied about the basic probability theory. Now I am studing about entropy from information theory point of view from Bose's Information Theory, Coding and Cryptography. I have not understood ...
0
votes
0answers
24 views

Does autocorrelation in time series make meaningless the conditional entropy?

The conditional entropy of ${\displaystyle Y}$ given ${\displaystyle X}$ is defined as $${\displaystyle \mathrm {H} (Y|X)\ =-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log {\frac {p(x,y)}{p(...
1
vote
1answer
34 views

A question about the information content in the entropy formula

One intuition in the entropy definition is that there is an inverse relationship between the information content of an event and its probability. This makes sense since learning an event which has a ...
1
vote
0answers
16 views

How can I show a mathematical proof of entropy in clasification tree? [closed]

I am trying to understand the splitting criteria in the classification tree. How can I show that for $p_1,p_2,..,p_n$ these functions attaining their maximum and minimum? $g(p_1,p_2,...,p_n) = Σp_i(1-...
1
vote
1answer
61 views

Does the entropy of a random variable change under a linear transformation?

Let $X$ be a random variable. If $Y=aX+b$, where $a,b \in \mathbb{R}$, is the entropy of $Y$ the same as the entropy of $X$?
3
votes
1answer
122 views

Information loss

Let's imagine we have some binary random variable $Y$ and some other continuous variable $X\in\mathbb{R}$ and we have some sample of size $n$. Suppose we want to determine the relationship between $Y$ ...
1
vote
1answer
36 views

Joint entropy of a bivariate Gamma probability density function

In Nadarajah and Kotz, 2009 (https://projecteuclid.org/journals/rocky-mountain-journal-of-mathematics/volume-39/issue-1/Four-Bivariate-Distributions-with-Gamma-Type-Marginals/10.1216/RMJ-2009-39-1-231....
1
vote
0answers
15 views

How to quantify information in a time series?

I have a time series of daily values for one year and I want to find out which slice of the time series contains maximum information considering a slice is of 150 values. I calculated Shannon's ...
1
vote
1answer
58 views

The correct way to form a PDF from probabilities of means (to calculate Entropy later)

The Question I have a probabilistic belief. I have 4 normal distributions, and I believe in each one a different amount. As follows: $$ P(M=ceramic)=0.9987 \\ P(M=aluminium)=0.0013 \\P(M=plastic)=0 \\ ...
2
votes
1answer
69 views

How the entropies of the random variables in conditional entropy affect its value?

We know that the conditional entropy $H(Y|X)\to 0$ as $X$ determines the value of $Y$. Now, I have the intuition of that $H(Y|X)<H(Y|Z)$ if $H(X)<H(Z)$. With this, and from the first statement, ...
6
votes
4answers
560 views

Why is entropy sometimes written as a function with a random variable as its argument?

Why do people sometimes write the entropy as a function on a random variable? For example, in some class notes I am seeing: $$ \sum_X Q(X) \log Q(X) = - H_Q(X) $$ I realize this is not to be taken as ...
5
votes
0answers
64 views

Copula entropy: calculation is borked?

I came across a pretty cool paper whose idea makes a lot of sense to me. Ma, Jian, and Zengqi Sun. "Mutual information is copula entropy." Tsinghua Science & Technology 16.1 (2011): 51-...
0
votes
1answer
44 views

How can I code the sequence of [1,2,3,4,5,6,6,5] in 2.5 or 3 bits?

If I calculate the entropy for the following sequence: [1,2,3,4,5,6,6,5] I get the entropy of 2.5 but I am wondering how can I actually do the encoding with 2.5 or 3 bits. Does it mean I need 3 bits ...
2
votes
0answers
15 views

Detecting behaviour change in a recommender system

Suppose I want to track a recommender system's live performance. The task isn't exactly to detect outliers, but to detect if the system started behaving differently, looking at the output only - an ...
0
votes
0answers
73 views

Why is the cross-entropy loss function used in logistic regression? [duplicate]

Just wanted to double check: I understand that we can't use the squared error cost function (as we do in linear regression) for the cost function of a logistic regression model as the sigmoid function ...
2
votes
0answers
52 views

Generalization of Burg's Maximum Entropy Theorem

Burg's Theorem characterizes the form of an entropy-maximizing time series, subject to constraints on the autocorrelation. More precisely, the theorem states that the autoregressive Gaussian process $...
0
votes
0answers
34 views

Differential entropy/LDDP with F-distributed data

I am working with a dataset where many variables are an F-distribution, particularly distributed like the black and gray curves in this graph (taken from Wikipedia). I am interested to calculated ...
2
votes
2answers
270 views

How can I prove that the conditional entropy of $X$ given $Y$ is $0$ if and only if $Y = f(X)$?

I want to show that: $$ H(X|Y) = 0 \iff Y=f(X) $$ Where $H(X|Y)$ is the average conditional entropy of the discrete random variable $X$ over all values of the discrete random variable $Y$, and $f$ is ...
1
vote
1answer
27 views

Correct notation for the probability of an event in entropy

I am looking at the formula of entropy on Wikipedia, where $P(X)$ is a probability mass function. \begin{equation} H(X) = -\sum_{i=1}^{n}P(x_i)log_bP(x_i) \end{equation} I got curious why they use ...
1
vote
0answers
41 views

Why is the Kullback-Leibler divergence defined with a negative sign?

I am aware of Gibbs' inequality, but I still want to know why the Kullback-Leibler divergence is defined with a negative sign. Here is my reasoning so far: Let $X$ be a Bernoulli random variable with $...

1
2 3 4 5
13