A mathematical quantity designed to measure the amount of randomness of a random variable.
0
votes
0answers
3 views
Correct number of grayscale levels when computing entropy of an image
I want to compute and to compare the entropy of images. Right now I am using images with 256 grayscale levels. But should the number of level not depend on the image size? For a really small image, ...
0
votes
0answers
16 views
Confusion about maximising vs minimising (relative) entropy terminology and methods
I'm studying rules of inference for updating from a prior probability distribution to a posterior. One method for doing this is by maximising entropy, subject to constraints. I'm reading papers like ...
1
vote
0answers
9 views
Is it possible to use Information Gain metric for CART?
I've been looking for an example of CART using Information Gain but haven't found one. This made me wonder if it was even possible, so I tried to train one manually (by hand) using the dataset below ...
0
votes
1answer
25 views
Question when using sklearn's DecisionTreeClassifier
sklearn's DecisionTreeClassifier is not behaving as I expected. From the following:
...
2
votes
0answers
24 views
How do we show the exponential distribution has maximal entropy on R+?
I’ve been looking around the internet, and having trouble finding a demonstration that the exponential distribution is maximal entropy on R+. I’d appreciate any points in the right direction.
0
votes
0answers
13 views
is there a measure for the roughness of a contour plot
There has to be a measure for the difference between "instantaneous" change of "energy" along a line in a space compared to averaged changed of energy along a line.
I could take a smooth surface in ...
0
votes
1answer
13 views
Difference of notation between cross entropy and joint entropy
Although it is clear to me, how the two concepts differs, it has been difficult for me to find a notation that would make it clear, to which type of entropy we refer.
From wikipedia, we can see that ...
0
votes
1answer
42 views
what is the meaning or intuition of entropy (from the point of view of reinforcement learning)
Can someone give an intuition of the concept 'entropy'?
I am reading maximum entropy inverse reinforcement learning and I wanted to ask what the meaning intuition of 'entropy' is.
I understand ...
0
votes
0answers
6 views
Does perplexity indicate how often my model gets it right?
This answer currently has the most votes and suggests
The perplexity of whatever you're evaluating, on the data you're evaluating it on, sort of tells you "this thing is right about as often as an ...
0
votes
1answer
45 views
How to measure how “good” or accurate a probability distribution is? Entropy, variance or what?
How can one measure the accuracy of the probability distribution of, say, a physical magnitude?
I know one good candidate is the entropy, which measures the amount of information one has about the ...
1
vote
0answers
39 views
How can I maximise binary cross entropy loss?
I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
1
vote
1answer
22 views
Why is the cross-entropy always more than the entropy?
I understand intuitively why cross-entropy is always bigger. However, could someone show that mathematically?
0
votes
0answers
17 views
central limit theorem in term of entropy
The usual central theorem uses iid samples.
Are there a generalizing theorem to non-iid samples, using the conditional entropy of each sample given previous ones ?
3
votes
1answer
32 views
Which result for normalized mutual information is correct?
I wanted to find the normalized mutual information to validate a clustering algorithm, but I've encountered two different values depending on the library I use.
In Python:
...
0
votes
1answer
27 views
Does downsampling decrease the entropy of the data?
Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$.
Can we say this downsampling ...
0
votes
0answers
31 views
On the meaning of mutual information and on how to test the convergence of an estimation
As the result of a Molecular Dynamics simulation, I have the time series of two variables, $X$ and $Y,$ and I am interested in computing the mutual information of these two random variables. I've ...
0
votes
0answers
19 views
Estimating a surprise of a word in context
What will be the best way to estimate the entropy/surprise of a word in a specific context?
Let's say to compare the surprise of:
context:
"I watched the movie in my"
word:
Computer
I ...
0
votes
1answer
75 views
NLP various probabilities estimators in nltk
I saw there are many types of probabilities in nltk:
MLE, ELE, Laplace, Heldout, KnereserNey, Lidstone, Random, WittenBel..
What is the exact difference between them and when should I use each?
My ...
1
vote
0answers
35 views
Entropy of a mixture of Gaussians
I need to estimate as fast and accurately as possible the differential entropy of a mixture of $K$ multivariate Gaussians:
$$
\mathcal{H}[q] = -\sum_{k=1}^K w_k \int q_k(\textbf{x}) \log \left[\sum_{...
0
votes
0answers
33 views
Is continuous mutual information the correct analogue of the discrete version?
I'm interested in the mutual information of two continuous random variables $X$ and $Y$. Shannon defined differential entropy as $h(X) = -\int p_X(x)\log p_X(x) dx$, where $p_X$ is the probability ...
2
votes
0answers
38 views
How to compare two point estimates between two samples
I have two samples, composed by independent observations and a set of features. Each sample can be represented as a binary matrix ($A_{ij} \in \{0, 1\}$) of observations x features. I want to find ...
0
votes
0answers
29 views
Test for Uniform Distribution
The data I have has 24 data points for each observation. Each data point can take value of either zero or one. I want to test whether the 24 values come from a (discrete) uniform distribution. What ...
1
vote
0answers
41 views
How to select value of k in K-NN when using Transfer Entropy?
I was reading about Transfer Entropy and I came across the estimators used to calculate TE, one of them being the Kraskov Estimator:
$ T_{X \rightarrow Y} = \frac{p(Y_{n+1}, Y(k)_{n}, X(l)_{n})*log(...
1
vote
0answers
21 views
How does Noise affect the results of Transfer Entropy?
I was reading about Transfer Entropy and came across this package: https://cran.r-project.org/web/packages/TransferEntropy/TransferEntropy.pdf
The code in the package:
...
0
votes
0answers
49 views
What is “Entropic Capacity”?
I found this term on the Keras blog website, quoted below
Your main focus for fighting overfitting should be the entropic capacity of your model --how much information your model is allowed to ...
1
vote
0answers
32 views
Priors in Shannon and Rényi entropies
[Note: Cross-posted at Math StackExchange]
I am new to information theory and currently working with Shannon and Rényi entropies. Given the pdf $p_{\theta}(x)$ of a random variable $x$, that is ...
1
vote
0answers
16 views
Naive Bayes: Understanding the Entropy equation
I am trying to understand the entropy equation:
-p1*log2(p1) - p2*log2(p2) - pn*log2(pn)
Specifically why do we multiply each log by the probability? In the ...
0
votes
1answer
92 views
Meaning and interpretation of Transfer Entropy
I am a first-year undergrad student and I have been reading about Transfer Entropy for my research. Although I understand the math behind I am not really sure what the value means. For example, I run $...
0
votes
2answers
73 views
Entropy or co-occurence matrix to compute the randomness of gray scale images?
I hope this is the right place to ask this question.
I have an algorithm that outputs gray scale images (not normalized). These images oftentimes contain a lot of random noise and sometimes ...
4
votes
1answer
547 views
What is the difference Cross-entropy and KL divergence?
Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference?
$$ H(P,Q) = -\sum_x P(x)\log Q(x) $$
$$ KL(P | Q) = \sum_{x} P(...
1
vote
0answers
29 views
Mutual information between continuous $Z$ and $g(Z)$ for differentiable $g$?
I have a continuous random variable $Z$ and a differentiable function $X = g(Z)$. Is the mutual information between $X$ and $Z$ necessarily $\infty$ or 0? Are there any examples of differentiable ...
0
votes
1answer
52 views
Computing joint entropy from marginal distributions
I have distributions of N random variables (supposed conditionally independent)
consequently, the joint distribution is the multiplication of all the distributions.
I want to compute the joint ...
2
votes
0answers
125 views
Can the differential entropy be negative infinity?
Define the (differential) entropy for density $f$ as
$$ H(f) :=-\int_{0}^{1} f(x) \log_{2}(f(x)) dx \, .$$
I am trying to find a Lebesgue measurable $f$ defined on $[0,1]$ such that $f\geq 0, \...
0
votes
1answer
28 views
Shannon entropy with regards to independent random variables
I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific ...
1
vote
1answer
385 views
Cross-entropy for comparing images
Suppose we have two greyscale images which are flattened to 1d arrays: $y=(y_1, y_2, \ldots, y_n)$ and $\hat{y} = (\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_n)$ with pixel values in $[0,1]$. How exactly ...
1
vote
0answers
18 views
Conditions Mutual Information and Confounding Effect
Given that conditional mutual information (CMI) I(A,B |C) is the information shared between A, and B given C, does this consider the confounding effect -if any - that C introduces? In other words, ...
7
votes
2answers
161 views
Interpretation of entropy for continuous distribution?
"Entropy" roughly captures the degree of "information" in a probability distribution.
For discrete distributions there is a far more exact interpretation: The entropy of a discrete random variable ...
7
votes
1answer
150 views
Distribution of $-\log f_X(X)$
Say, $X \in \mathbb{R}^n$ (with $n > 1$) has a density $f_X(x)$. What can we say about the distribution of
$$
Y = -\log f_X(X)?
$$
1
vote
1answer
357 views
How to compute joint entropy of high-dimensional data?
Normally, I compute the (empirical) joint entropy of some data, using the following code:
...
1
vote
0answers
191 views
Trying to implement the Jensen-Shannon Divergence for Multivariate Gaussians
Given two multivariate Gaussian distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them.
...
0
votes
0answers
13 views
Calculating conditional mutual information for textual features
I am reading this paper regarding the usage of conditional mutual information in textual data. I am unable to understand or see how cmi between certain features would enrich a classification algorithm ...
2
votes
0answers
42 views
How to prove subadditivity of joint entropy?
I've been kind of stuck on this exercise for the last hours.
Heres everything I know and everything I'm allowed to use to prove it:
Let $X= \{w_1, \ldots, w_n \}$ and $Y = \{v_1, \ldots, v_n \}$ be ...
2
votes
1answer
90 views
Entropy of the beta-binomial compound distribution
I have a generative process as follows:
$$
x \mid \alpha \sim \textsf{Beta}\left (\alpha,\beta \right) \\
y \mid x \sim \textsf{Bernoulli}(x).
$$
How does one go about calculating the Entropy of ...
2
votes
2answers
251 views
Entropy of a function of independent random variables
Suppose I have an operator (function) $f(\cdot)$ which takes three arguments $x,y,z$ all of which are independent random variables, and all of which I have access to the probability mass function (...
0
votes
0answers
23 views
Entropy of a factorised joint distribution
Suppose I have three discrete random variables $X, Y$ and $Z$.
Their joint distribution factorises as so:
$$
P(X,Y,Z) = P(X)P(Y)P(Z)
$$
i.e. they are fully independent variables.
Now suppose I want ...
0
votes
0answers
9 views
Entropy of a binary matrix distributed according to an Indian Buffet Process?
Given that a probability mass function gives the probability that a discrete random variable is exactly equal to some value, and that we do not possess a PMF for the Indian Buffet Process - is it ...
1
vote
1answer
117 views
Entropy of a matrix with Bernoulli distributed (binary entries) row-vectors
The entropy $H[x]$ of a Bernoulli distributed binary random variable $x$ is given by :
$$
H[x]=−θlnθ−(1−θ)ln(1−θ)
$$
where
$$
p(x=1∣θ)= \theta \\
p(x=0∣θ)=1−θ
$$
Now, suppose I have a vector as so:
...
0
votes
0answers
59 views
How to Calculate the multi scale approximate entropy for a financial time series
I want to calculate the multiscale approximate entropy for the financial time series. The code which I got to calculate approximate entropy is as follows:
approx_entropy(ts, edim = 2, r = 0.2*sd(ts), ...
-1
votes
2answers
84 views
Measure of neatness of a file tree [closed]
I'm working on a way to calculate the "neatness" of any file system tree.
Features I'm considering include:
depth of directory from root
neatness of child directories
number of files/child ...
2
votes
1answer
86 views
Was logistic regression based on Boltzmann distribution from statistical mechanics?
Ever since seeing the logistic distribution for the first time many years ago, I always thought of it as an application of the Boltzmann distribution. Whoever developed it may had seen the Boltzmann ...
