Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
603
questions
0
votes
0answers
6 views
Determining history length in Transfer Entropy
Supposing I have a time series observation corresponding to 5 sensors, and I wish to find the causation(Transfer entropy) between all pairs of nodes, how do I go about choosing the history length ...
2
votes
1answer
30 views
Measurement of regularity of events
I am dealing with events generated by different processes. My data includes time stamps when the events occur, so I am trying to differentiate the processes in regular or irregular categories or ...
0
votes
0answers
4 views
How to calculate the uncertainty of a variable explained by other variables
I'm working on implementing a method from a paper for solving the FRn-k problem for a specific type of biological data.
[FRn−k]: Given an initial set of n features, find the subset with k<n
...
2
votes
1answer
35 views
Differential Entropy with negative support
Differential Entropy Estimation
Hello everyone, I am estimating differential entropy from a continuous distribution of a property of materials. The distributions on the picture are Gaussian Mixtures. ...
3
votes
1answer
124 views
Is there a name for this in statisitics? [closed]
Edit: Ignore this Most of this doesn't make sense and is beyond edits but its too late to delete. If I make too many edits I could be kicked out of this sub like I was in math stack exchange.
I don't ...
3
votes
1answer
39 views
Mutual information is not zero for independent variables and negative for weakly dependency
To the best of my knowledge, mutual information (MI) is zero if and only if the variables are independent.
I have simulated copula data and computed the ...
0
votes
0answers
13 views
Negative Tsallis entropy?
I am building (many years later !) on this SO question: https://stackoverflow.com/questions/22461241/tsallis-entropy-for-continuous-variable-in-r
I should make clear that I'm a stats newbie, exploring ...
4
votes
1answer
44 views
How are entropy and Gini Impurity related?
I know the differences between entropy and Gini impurity and why we use Gini in order to construct trees. But I would like to find some relation between those two measures. It leads me to one ...
0
votes
0answers
26 views
Lower bound on conditional entropy when conditioning on event
In Lemma 3.8 of this paper, the following is shown for jointly distributed random variables $Z$, $W$, where $Z$ takes values in $\{0,1\}^n$, and some event $\mathcal{E}$:
$$
H[ Z \mid W ] \ge n - d\ \...
0
votes
0answers
25 views
Gradient with respect to a probability distribution
I'm trying to understand the worked propositions on this paper (reinforcement learning): https://arxiv.org/abs/2007.02832
The authors formulate this objective:
$\hat{g}^*=\arg\max_{\hat{g}\in B}\...
2
votes
1answer
78 views
Base of the logarithm used for Shannon Entropy when considering a vocabulary
If we want to calculate the shannon entropy of a text, considering that the elementary symbols aren't characters but words, will we still use a base-2 logarithm ? Or will we use a base-n logarithm ...
1
vote
0answers
34 views
KL divergence from a family of distributions
The KL divergence between two probability measures $q,p$ is $$KL(q(x)||p(x)) = - \int_{\mathcal{X}} q(x) \log\frac{p(x)}{q(x)} dx.$$
What's the KL divergence between a measure and a set of measures? ...
1
vote
1answer
51 views
Does comparing Shannon Entropy between categorical variables makes any sense?
I want to present a report that highlights the variables with lowest information, to emphasize that some action must be taken by the department that controls our data source. I've applied Shannon ...
0
votes
1answer
33 views
Entropy of gaussian mixture
Does the entropy of a gaussian mixture depend on its means? It is not the case for a single Gaussian and when the components of the mixture are far spread out, we can approximate the entropy by a ...
0
votes
0answers
34 views
What is extropy?
While searching for the answer to a statistics problem, I came across the term extropy, which was - and to a large extent still is - unfamiliar to me. A quick google search revealed that it is the &...
0
votes
0answers
4 views
Choosing appropriate values for $m$ and $r$ in approximate entropy
Is there a standard way to determine optimal values for $m$ and $r$ in approximate entropy, maybe based on some quantitative criteria or statistics of the time series?
Wikipedia defines
The value of $...
0
votes
0answers
41 views
Example of mutual information calculated with R packages in the case of continuous normal variables
Nassim Taleb mentions mutual information as an alternative to correlation, given the tendency of correlation to be very different from zero even in draws from a normal distribution in a simulation.
I ...
0
votes
1answer
63 views
How can we determine Conditional Mutual Information based on multiple conditions
Let X,Y,Z be jointly distributed, the conditional mutual information is defined as :-
Similarly if we have 4 (or more variables) say X,Y,Z,W then how can we determine the conditional mutual ...
0
votes
0answers
19 views
Scale independent entropy?
Let $i \mapsto f(i)$ denote a probability distribution on a discrete space $I_n=\{1,\dots,n\}$. It is common to use Shannon entropy $H(n) = -\sum f(i)\log f(i)$ as a measure of uncertainty, where I ...
2
votes
1answer
24 views
Can anyone suggest minimum and maximum values that the relative entropy/KL divergence can range when computed from two time-series sequences?
Can anyone suggest minimum and maximum values that the relative entropy/KL divergence can range when computed from two time-series sequences?
I have calculated the relative entropy for two time-series ...
0
votes
2answers
64 views
Is it true that entropy estimation is meaningless if samples are not i.i.d.?
Let $X$ be a discrete-support stochastic variable. Information entropy is a number defined as
$$H(X)=-\sum_{n} p\left(x_{n}\right) \log p\left(x_{n}\right) \geq 0$$
Let $\hat{H}$ be an estimation of $...
2
votes
0answers
28 views
Why does Kullback–Leibler divergence measure information loss when approximating a probability distribution?
I've encountered a sentence:
In information theory, Kullback–Leibler divergence is regarded as a measure of the information lost when probability distribution Q is used to approximate a true ...
0
votes
0answers
25 views
Showing that conditional entropies $H(X \mid Y)$ and $H(Y \mid X)$ are equal for delta distributions
I need some help with the following problem.
What does it mean for the joint probability distribution of two random variables $x$ and $y$, $p(x,y)$ , to be equal to $\delta (x,y)$ (Kronecker delta)?...
1
vote
0answers
15 views
How am I getting this entropy calculation wrong?
I am following the entropy calculation in 1a here.
Shouldn't the solution for H(Edible | Odor = 1 or 3) be like this?
$H = -\frac{3}{6}\log_2{\frac{3}{6}} = \frac{1}{2}$
My reasoning is that we ...
2
votes
1answer
29 views
How can I show that the entropy of a function of random variables cannot be greater than their joint entropy?
Given the discrete random variables $X,Y,$ and $Z=f(X,Y)$, where $f$ is some function, how can I show that:
$$
H(Z) \leq H(X,Y)
$$
With equality if the function $f$ is invertible?
2
votes
1answer
57 views
Controlling the entropy of a distribution
What is the best way to control the entropy of a categorical probability distribution?
I have a categorical probability distribution $p_1, p_2, \dots, p_K$, for $K$ small-ish ($< 1000$). Assume ...
0
votes
0answers
23 views
How do we get from entropy to KL divergence in this paper?
I'm reading through Regularizing Neural Networks By Penalizing Confident Output Distributions and I'm stuck on the equation in section 3.2. It's not clear to me at all that the self-entropy of the ...
1
vote
1answer
116 views
Intuition on Wasserstein Distance
I've been trying to familiarize myself with the Wasserstein distance and saw this answer on StackExchange by @antike that at first made a lot of sense, but then it didn't (to me, of course).
In the ...
3
votes
0answers
221 views
Geometric distribution and entropy
According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{
\mu} $ ...
1
vote
0answers
32 views
Generate data that matches a frequency distribution while preserving the original spatial structure
I am dealing with a 3D array containing values representing the "importance" of each voxel. For my analysis, I would like to synthesize n new arrays from my original array to have a ...
0
votes
1answer
70 views
Difference between mutual and conditional information
I have studied about the basic probability theory. Now I am studing about entropy from information theory point of view from Bose's Information Theory, Coding and Cryptography.
I have not understood ...
0
votes
0answers
24 views
Does autocorrelation in time series make meaningless the conditional entropy?
The conditional entropy of ${\displaystyle Y}$ given ${\displaystyle X}$ is defined as
$${\displaystyle \mathrm {H} (Y|X)\ =-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log {\frac {p(x,y)}{p(...
1
vote
1answer
34 views
A question about the information content in the entropy formula
One intuition in the entropy definition is that there is an inverse relationship between the information content of an event and its probability. This makes sense since learning an event which has a ...
1
vote
0answers
16 views
How can I show a mathematical proof of entropy in clasification tree? [closed]
I am trying to understand the splitting criteria in the classification tree. How can I show that for $p_1,p_2,..,p_n$ these functions attaining their maximum and minimum?
$g(p_1,p_2,...,p_n) = Σp_i(1-...
1
vote
1answer
61 views
Does the entropy of a random variable change under a linear transformation?
Let $X$ be a random variable. If $Y=aX+b$, where $a,b \in \mathbb{R}$, is the entropy of $Y$ the same as the entropy of $X$?
3
votes
1answer
122 views
Information loss
Let's imagine we have some binary random variable $Y$ and some other continuous variable $X\in\mathbb{R}$ and we have some sample of size $n$.
Suppose we want to determine the relationship between $Y$ ...
1
vote
1answer
36 views
Joint entropy of a bivariate Gamma probability density function
In Nadarajah and Kotz, 2009 (https://projecteuclid.org/journals/rocky-mountain-journal-of-mathematics/volume-39/issue-1/Four-Bivariate-Distributions-with-Gamma-Type-Marginals/10.1216/RMJ-2009-39-1-231....
1
vote
0answers
15 views
How to quantify information in a time series?
I have a time series of daily values for one year and I want to find out which slice of the time series contains maximum information considering a slice is of 150 values. I calculated Shannon's ...
1
vote
1answer
58 views
The correct way to form a PDF from probabilities of means (to calculate Entropy later)
The Question
I have a probabilistic belief. I have 4 normal distributions, and I believe in each one a different amount. As follows:
$$
P(M=ceramic)=0.9987 \\ P(M=aluminium)=0.0013 \\P(M=plastic)=0 \\ ...
2
votes
1answer
69 views
How the entropies of the random variables in conditional entropy affect its value?
We know that the conditional entropy $H(Y|X)\to 0$ as $X$ determines the value of $Y$. Now, I have the intuition of that $H(Y|X)<H(Y|Z)$ if $H(X)<H(Z)$. With this, and from the first statement, ...
6
votes
4answers
560 views
Why is entropy sometimes written as a function with a random variable as its argument?
Why do people sometimes write the entropy as a function on a random variable?
For example, in some class notes I am seeing:
$$ \sum_X Q(X) \log Q(X) = - H_Q(X) $$
I realize this is not to be taken as ...
5
votes
0answers
64 views
Copula entropy: calculation is borked?
I came across a pretty cool paper whose idea makes a lot of sense to me.
Ma, Jian, and Zengqi Sun. "Mutual information is copula entropy." Tsinghua Science & Technology 16.1 (2011): 51-...
0
votes
1answer
44 views
How can I code the sequence of [1,2,3,4,5,6,6,5] in 2.5 or 3 bits?
If I calculate the entropy for the following sequence:
[1,2,3,4,5,6,6,5]
I get the entropy of 2.5 but I am wondering how can I actually do the encoding with 2.5 or 3 bits. Does it mean I need 3 bits ...
2
votes
0answers
15 views
Detecting behaviour change in a recommender system
Suppose I want to track a recommender system's live performance. The task isn't exactly to detect outliers, but to detect if the system started behaving differently, looking at the output only - an ...
0
votes
0answers
73 views
Why is the cross-entropy loss function used in logistic regression? [duplicate]
Just wanted to double check:
I understand that we can't use the squared error cost function (as we do in linear regression) for the cost function of a logistic regression model as the sigmoid function ...
2
votes
0answers
52 views
Generalization of Burg's Maximum Entropy Theorem
Burg's Theorem characterizes the form of an entropy-maximizing time series, subject to constraints on the autocorrelation. More precisely, the theorem states that the autoregressive Gaussian process $...
0
votes
0answers
34 views
Differential entropy/LDDP with F-distributed data
I am working with a dataset where many variables are an F-distribution, particularly distributed like the black and gray curves in this graph (taken from Wikipedia).
I am interested to calculated ...
2
votes
2answers
270 views
How can I prove that the conditional entropy of $X$ given $Y$ is $0$ if and only if $Y = f(X)$?
I want to show that:
$$
H(X|Y) = 0 \iff Y=f(X)
$$
Where $H(X|Y)$ is the average conditional entropy of the discrete random variable $X$ over all values of the discrete random variable $Y$, and $f$ is ...
1
vote
1answer
27 views
Correct notation for the probability of an event in entropy
I am looking at the formula of entropy on Wikipedia, where $P(X)$ is a probability mass function.
\begin{equation}
H(X) = -\sum_{i=1}^{n}P(x_i)log_bP(x_i)
\end{equation}
I got curious why they use ...
1
vote
0answers
41 views
Why is the Kullback-Leibler divergence defined with a negative sign?
I am aware of Gibbs' inequality, but I still want to know why the Kullback-Leibler divergence is defined with a negative sign. Here is my reasoning so far:
Let $X$ be a Bernoulli random variable with $...