# Questions tagged [entropy]

A mathematical quantity designed to measure the amount of randomness of a random variable.

468 questions
Filter by
Sorted by
Tagged with
19 views

### Does Entropy values enough creterion to determine the nature of any probability distribution?

Entropy is the measure of randomness of a process or variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i))$, Now there are ...
13 views

### Decision trees: maximizing information gain vs. minimizing conditional entropy?

Information gain is defined as $$IG(T, a) = H(T) - H(T|a),$$ where $H(T|a)$ is the conditional entropy of $T$ given attribute $a$, and $H(T)$ is the prior entropy of our dataset before we test out the ...
32 views

### “Any meaning to the concept of ‘Self Mutual Information?”

** “Any meaning to the concept of ‘Self Mutual Information?” ** A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
50 views

### Two meanings of entropy?

In the world of physics, entropy seems to mean something different from stats + information theory world. So, I assumed that there are two definitions to the word entropy. Indeed, I looked up how ...
6 views

### State of the art library for estimating differential entropy from real data

I am looking for a state of the art library for estimating differential entropy from finite samples. In an ideal world, it would have the following features: Work with real-valued multi-dimensional ...
17 views

### Transfer entropy and the sign of relationship

Transfer entropy (TE) measures the amount of directed transfer of information between two random processes, $X$ and $Y$. In other words, through the TE we can quantify the amount of information that ...
7 views

### Entropy in mole fraction distribution

I am studying how mole fractions are distributed in real life fuels (summation of mole fractions for any fuel=1, always). In this regard, I was trying to see if I can correlate my observations to ...
29 views

### Entropy of Joint Distribution of two IID Random Variables

I have two identical probability distributions $p_1(x) = p_2(x)\ \forall x$ and wish to construct a joint distribution between them, i.e. $p(x_1,x_2) = p_1(x_1)\cdot p_2(x_2)$. Supposing I know the ...
3k views

### Who is Gail Gasram?

Several places (1, 2, 3) quote someone named Gail Gasram as saying "Nothing is random, only uncertain" but a Google search turns up no info, just more places with this quote! Generally, it's in the ...
5 views

### Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?

I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long. I'm trying to ...
38 views

66 views

### Expanding initial sample when the result isn't significant

This inspiring answer describes a variant of hypothesis, and I want to analysis its property further. Basically, it considers a two-sided test and interprets the $p$-value as a measure of how strong ...