Questions tagged [entropy]
A mathematical quantity designed to measure the amount of randomness of a random variable.
583
questions
1
vote
0answers
16 views
Geometric distribution and entropy
According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{
\mu} $ ...
1
vote
0answers
9 views
Generate data that matches a frequency distribution while preserving the original spatial structure [closed]
I am dealing with a 3D array containing values representing the "importance" of each voxel. For my analysis, I would like to synthesize n new arrays from my original array to have a ...
0
votes
1answer
41 views
Difference between mutual and conditional information
I have studied about the basic probability theory. Now I am studing about entropy from information theory point of view from Bose's Information Theory, Coding and Cryptography.
I have not understood ...
0
votes
0answers
23 views
Does autocorrelation in time series make meaningless the conditional entropy?
The conditional entropy of ${\displaystyle Y}$ given ${\displaystyle X}$ is defined as
$${\displaystyle \mathrm {H} (Y|X)\ =-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log {\frac {p(x,y)}{p(...
1
vote
1answer
28 views
A question about the information content in the entropy formula
One intuition in the entropy definition is that there is an inverse relationship between the information content of an event and its probability. This makes sense since learning an event which has a ...
0
votes
0answers
11 views
How can I show a mathematical proof of entropy in clasification tree?
I am trying to understand the splitting criteria in the classification tree. How can I show that for $p_1,p_2,..,p_n$ these functions attaining their maximum and minimum?
$g(p_1,p_2,...,p_n) = Σp_i(1-...
0
votes
1answer
21 views
Does the entropy of a random variable change under a linear transformation?
Let $X$ be a random variable. If $Y=aX+b$, where $a,b \in \mathbb{R}$, is the entropy of $Y$ the same as the entropy of $X$?
2
votes
0answers
68 views
+50
Information loss
Let's imagine we have some binary random variable $Y$ and some other continuous variable $X\in\mathbb{R}$ and we have some sample of size $n$.
Suppose we want to determine the relationship between $Y$ ...
1
vote
1answer
32 views
Joint entropy of a bivariate Gamma probability density function
In Nadarajah and Kotz, 2009 (https://projecteuclid.org/journals/rocky-mountain-journal-of-mathematics/volume-39/issue-1/Four-Bivariate-Distributions-with-Gamma-Type-Marginals/10.1216/RMJ-2009-39-1-231....
1
vote
0answers
12 views
How to quantify information in a time series?
I have a time series of daily values for one year and I want to find out which slice of the time series contains maximum information considering a slice is of 150 values. I calculated Shannon's ...
1
vote
0answers
44 views
The correct way to form a PDF from probabilities of means (to calculate Entropy later)
The Question
I have a probabilistic belief. I have 4 normal distributions, and I believe in each one a different amount. As follows:
$$
P(M=ceramic)=0.9987 \\ P(M=aluminium)=0.0013 \\P(M=plastic)=0 \\ ...
2
votes
1answer
40 views
How the entropies of the random variables in conditional entropy affect its value?
We know that the conditional entropy $H(Y|X)\to 0$ as $X$ determines the value of $Y$. Now, I have the intuition of that $H(Y|X)<H(Y|Z)$ if $H(X)<H(Z)$. With this, and from the first statement, ...
5
votes
4answers
517 views
Why is entropy sometimes written as a function with a random variable as its argument?
Why do people sometimes write the entropy as a function on a random variable?
For example, in some class notes I am seeing:
$$ \sum_X Q(X) \log Q(X) = - H_Q(X) $$
I realize this is not to be taken as ...
1
vote
0answers
33 views
Copula entropy: calculation is borked?
I came across a pretty cool paper whose idea makes a lot of sense to me.
Ma, Jian, and Zengqi Sun. "Mutual information is copula entropy." Tsinghua Science & Technology 16.1 (2011): 51-...
0
votes
1answer
40 views
How can I code the sequence of [1,2,3,4,5,6,6,5] in 2.5 or 3 bits?
If I calculate the entropy for the following sequence:
[1,2,3,4,5,6,6,5]
I get the entropy of 2.5 but I am wondering how can I actually do the encoding with 2.5 or 3 bits. Does it mean I need 3 bits ...
2
votes
0answers
14 views
Detecting behaviour change in a recommender system
Suppose I want to track a recommender system's live performance. The task isn't exactly to detect outliers, but to detect if the system started behaving differently, looking at the output only - an ...
0
votes
0answers
26 views
Why is the cross-entropy loss function used in logistic regression? [duplicate]
Just wanted to double check:
I understand that we can't use the squared error cost function (as we do in linear regression) for the cost function of a logistic regression model as the sigmoid function ...
2
votes
0answers
41 views
Generalization of Burg's Maximum Entropy Theorem
Burg's Theorem characterizes the form of an entropy-maximizing time series, subject to constraints on the autocorrelation. More precisely, the theorem states that the autoregressive Gaussian process $...
0
votes
0answers
7 views
Differential entropy/LDDP with F-distributed data
I am working with a dataset where many variables are an F-distribution, particularly distributed like the black and gray curves in this graph (taken from Wikipedia).
I am interested to calculated ...
1
vote
2answers
58 views
How can I prove that the conditional entropy of $X$ given $Y$ is $0$ if and only if $Y = f(X)$?
I want to show that:
$$
H(X|Y) = 0 \iff Y=f(X)
$$
Where $H(X|Y)$ is the average conditional entropy of the discrete random variable $X$ over all values of the discrete random variable $Y$, and $f$ is ...
1
vote
1answer
25 views
Correct notation for the probability of an event in entropy
I am looking at the formula of entropy on Wikipedia, where $P(X)$ is a probability mass function.
\begin{equation}
H(X) = -\sum_{i=1}^{n}P(x_i)log_bP(x_i)
\end{equation}
I got curious why they use ...
1
vote
0answers
33 views
Why is the Kullback-Leibler divergence defined with a negative sign?
I am aware of Gibbs' inequality, but I still want to know why the Kullback-Leibler divergence is defined with a negative sign. Here is my reasoning so far:
Let $X$ be a Bernoulli random variable with $...
0
votes
1answer
15 views
Is the generalized entropy of certain events alway null?
In their paper: Novel Decompositions of Proper Scoring Rules for Classification, Kull and Flach wrote in section 2.2 Divergence, Entropy and Properness that when $y$ is the true class $d(p,y)=\phi(p,...
2
votes
0answers
35 views
Name for a particular family of distributions on unit simplex
Let $\Delta^n$ be the unit simplex, and let $\alpha\ge 0$ be a parameter. Is there a standard name for the following probability density function supported on $\Delta^n$?
$$f_{\alpha}(p)=C(\alpha)e^{-\...
3
votes
2answers
50 views
How to conceptualise statistical entropy
Preamble
There's this great YouTube video explaining the entropy of a probability distribution using the idea of "what is the minimum number of questions I need to ask?".
So as an example, ...
0
votes
0answers
28 views
Measuring entropy/concentration in multivariate datasets
I am interested in measuring how concentrated or widely dispersed a certain binary property is among a population, which is defined by a number of categorical variables.
For instance, let's say I have ...
0
votes
0answers
13 views
Change in Shannon Entropy in Markov Chain Process
What are the known results on the change in Shannon entropy $\Delta H_{k} = H(\vec{p}_{k}) - H(\vec{p}_{k-1})$ of the $k$-th step in a process governed by a finite state discrete time Markov chain ...
1
vote
1answer
38 views
Does logit transformation of information entropy values make sense?
I have a vector of information entropy values that range between 0 and 1 which I want to explain with some explanatory variables.
I realized that the distribution of the entropy values in my dataset ...
0
votes
0answers
23 views
Relationship between log-likelihood function and entropy (instead of cross-entropy)
Negative log-likelihood function
$$ -\ln f(X\mid\Theta)=-\int_{-\infty}^{\infty}\ln
f(x\mid\Theta) \, \mathrm{d}x$$
Differential entropy
$$ h(X) = -\int_{-\infty}^{\infty} f(x) \ln f(x) \, \mathrm{d}...
0
votes
1answer
38 views
Constructing a joint distribution from pairwise marginals
Consider a set of random variables $\{X_i\}$ with joint pdf $f(x_1 ... x_n)$. Given the marginal pdfs $f_i(x_i)$, we can construct a joint distribution
$$g(x_1 ... x_n) = \prod_i f_i(x_i)$$
which has ...
2
votes
1answer
22 views
Information Entropy Problem
I cannot figure out this simple entropy problem and it is driving me crazy!
From McElreath's Statistical Rethinking:
Imagine instead 5 buckets and a pile of 10 individually numbered pebbles. You stand ...
2
votes
1answer
35 views
“Information” Correlation
(Let $X$ and $Y$ be random variables, sufficiently nice for my question to make sense.)
$$
\text{Correlation}
$$
$$
\rho(X, Y) = \dfrac{\text{cov}(X, Y)}{\sqrt{\text{var}(X)}\sqrt{\text{var}(Y)}}
$$
...
2
votes
1answer
63 views
Information gain of the root node
Recently I saw this question and answer as attached in following image
Anyone can add details how this solution achieved?
1
vote
0answers
33 views
Can a likelihood's relative entropy be related to its predictive accuracy?
Suppose I have some prior $\pi(\theta)$, from which I draw $N$ samples, each having parameter $\theta_i$. These $\theta_i$'s are known to me. Suppose that one of these samples (unknown to me which) ...
0
votes
0answers
15 views
Bounds on the difference between entropies of two distributions?
Have there been any results regarding bounds on $\left| H(\hat{\pi})-H(\pi) \right|$ where $\hat{\pi}$ and $\pi$ are the empirical and true probability distribution?
0
votes
1answer
50 views
Clarification of variational autoencoders
I've been spending almost a month trying to understand VAE. I was reading a bunch of tutorials, and first it made sense, and seemed straightforward. Then I was experimenting with it, it produced weird ...
0
votes
1answer
15 views
Measure of segregation that takes population-level proportions into account (modified entropy/Theil index)
I am interested in a measure of segregation which allows for multiple groups and is maximized when group levels reflect the population as a whole (rather than equality of group size).
The best I have ...
1
vote
0answers
11 views
CNN: quantitatively evaluate the activation of a filter
When an image is fed into a CNN, it would pass through different layer of different filters. The visualization of these filters look like:
Here we can safely claim that the 1st filter is more ...
0
votes
0answers
5 views
Entropy of multivariate Pearson Type II
I was checking the entropy of multivariate Pearson Type II for maximizing Renyi entropy.
It would be great if someone came up with this question or advise some reliable sources.
0
votes
1answer
24 views
Minimising KL divergence between two distributions
say we want to approximate a distribution $p(x)$ with $q(x|\theta)$. We do not know the distribution $p(x)$ but we can draw samples from $p(x)$. The KL divergence between the two distributions is
$$
\...
3
votes
1answer
72 views
Why doesn't the entropy decrease with an increasing number of observations?
I'm trying to think more about entropy.
I have the following toy example:
Consider a coin flip.
Case 1:
I think p_h = 0.5
The entropy of this is 0.5 ln(0.5) x 2 = ln(0.5)
Case 2:
I don't know what p_h ...
3
votes
0answers
34 views
Statistically, is it effective to design exams to get harder as you answer correctly?
I think we have all taken exams where as you answer more questions correctly, the exam gets harder. Intuitively, this is obvious why...If an IQ test had questions for 3rd graders, everyone would get ...
0
votes
0answers
15 views
Information gain for unequally sized time series
I have some time series data, $\mathbf{x}_{1:T} = \{ x_1, \dots, x_T \}$ where the observation at time $t$, $X_t$, is a continuous random variable. Let $Y_t$ denote a discrete random variable at time $...
1
vote
0answers
25 views
Is there a metric to identify whether samples are uniformly distributed when number of samples is small?
Suppose I have a small number of samples drawn from an unknown distribution $\{X_1,X_2,...,X_n\}$, where $0\le X_i \le L$, and $3\le n \le10$. I want to identify a metric to understand how far these ...
1
vote
0answers
16 views
entropy regularization in generative model
I am wondering if it is possible to use entropy as a regularization in a generative model. For example, in the conjugate model where $x_i \in X$ is observed data and generated from a Normal ...
2
votes
1answer
79 views
Log probabilities versus squared probabilities (entropy vs Gini)
The advantage of log probabilities over direct probabilities, as discussed here and here, is that they make numerical values close to $0$ more easy to work with. (my question, instead of the links, ...
1
vote
0answers
35 views
When would you use purity as a measure of external validity over entropy? [closed]
This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
0
votes
0answers
16 views
How does Information Gain work?
I know the algorithm upon which the information gain is based. But how are we sure that after a split, its entropy will always decrease i.e. on dividing the data into smaller chunks, the entropy will ...
1
vote
1answer
68 views
Maximum Entropy Discrete Distribution
In Pattern Recognition and Machine Learning the author uses Lagrange multipliers to find the discrete distribution with maximum entropy.
Entropy is defined by;
$$H=-\sum_i p(x_i)\ln(p(x_i))$$
and the ...
0
votes
0answers
11 views
What is the effect of number of points on calculation of mutual information between two variables?
I was trying to calculate mutual information between two variables as follows:
mutual_info_score(x,y)
This essentially creates a 2d histogram and evaluates the ...
