A mathematical quantity designed to measure the amount of randomness of a random variable.

learn more… | top users | synonyms

0
votes
0answers
6 views

Kullback-Leibler divergence with sample data likelihood

I'm trying to get my head around the KB divergence in the context of the sample likelihood under two competing hypotheses, one optimal $H_0$ and one suboptimal $H_1$. Roughly speaking, I want to see ...
3
votes
0answers
19 views

Maximum entropy distribution of a proportion with known mean and variance? Is it a beta?

Given a proportion and its standard error, what distributional assumption minimizes assumptions/maximizes entropy? Is it the beta (and can I use the method of moments to estimate its parameters)? Or ...
0
votes
0answers
10 views

Definition of Conditional Entropy in Paper

I was reading this paper: InfoGAN. On page 4, in the first 2 lines of equations, I was wondering why the conditional entropy $$H(c|G(z,c) ) = -\mathbb{E}_{x \sim G(z,c)} [\mathbb{E}_{c' \sim P(c|x)} ...
1
vote
1answer
50 views

Tensorflow Cross Entropy for Regression?

Does cross-entropy cost make sense in the context of regression? (as opposed to classification) If so, could you give a toy example through tensorflow and if not, why not? I was reading about cross ...
1
vote
1answer
48 views

Shannon entropy and does missing data affect output

So I am doing analysis based on this paper: http://www.bioline.org.br/pdf?se10001 The author uses Shannon entropy in one part to calculate weights and he uses data which does not come with missing ...
0
votes
0answers
12 views

Weighting average similarity/correlation values by number of correlations?

I have a dataset containing correlations and other similarity measures between musical performances of which the performer and his nationality etc are known. Now i want to analyse that dataset; ...
0
votes
1answer
16 views

How to take random draws of a low-entropy “meta-random” distribution

I'm working on a simulation study, and for it I'd like to be able to generate random draws from a random multivariate distribution. I'm looking for something pretty chaotic, in the sense that it's ...
0
votes
0answers
13 views

Using differential entropy for characterizing a number

I have to come up with a score representing the information related to a number pulled from a a text (IM text). When correctly identified the numbers in our Ims follow a distribution composed of two ...
0
votes
1answer
45 views

kullback leibler divergence of empirical density and fitted density in R

I have a vector and I used Pareto and Log-normal distribution to get a predicted density function. I am wondering is there any easy ways to calculate the KLD between the true distribution and fitted ...
0
votes
0answers
16 views

How to deal with missing data when calculating Information Gain

While working on a neural network for classification problem I'm dealing with huge number of possible features and information gain seems like a good way to narrow them down (there are hundreds of ...
0
votes
0answers
9 views

How to calculate entropy for a specific attribute?

This is super simple but I'm learning about decision trees and the ID3 algorithm. I found a website that's very helpful and I was following everything about entropy and information gain until I got ...
3
votes
1answer
59 views

What happens if I flip targets and predictions in cross-entropy?

When we compute the cross-entropy within the machine learning context, we use the following formula: $$ CE(t, p) = -\sum_{i=1}^{N} t_i \ \log(p_i) $$ Where $t$ is the target class probability, and $...
12
votes
0answers
102 views

What does the Akaike Information Criterion (AIC) score of a model mean?

I have seen some questions here about what it means in layman terms, but these are too layman for for my purpose here. I am trying to mathematically understand what does the AIC score mean. But at ...
6
votes
1answer
369 views

Interpretation of the entropy with a coding length?

There is this interpretation of the entropy $-\sum_i p_i \log_2 p_i$ as the average length (in bits) per character when using an optimal encoding of a message. Now, if we use the simple 3-letter case ...
1
vote
2answers
44 views

Transfer entropy value between 0 and 1

Given two variables, X and Y, there is a way of obtaining a Mutual Information value between 0 and 1 by: MI_normalised=MI_original/sqrt(H(X)*H(Y)); where H(X) ...
0
votes
1answer
29 views

Interpreting the entropy of a Dirichlet distribution

I was looking for a measure to interpret the "spikiness" of categorical histograms. So, if it becomes unnaturally skewed towards a certain value at a given time, I want a metric that will show some ...
0
votes
0answers
3 views

Efficient way to compute Entropy involving three variables

Let $X$, $Y$ and $Z$ be three discrete random variables, such that $X$ and $Y$ are mutually independent and identically distributed but $Z$ is dependent on $X$ and $Y$. Information entropy is ...
3
votes
0answers
27 views

Gibbs entropy and Shannon entropy

The two formulations seem identical to me: $H(x) = \sum p(x) log(1/p(x))$ why would the equation be attributed to Shannon rather than Gibbs (in the context of information)?
1
vote
0answers
10 views

Calculating joint entropy of two variables having null values

I calculated the joint entropy of two random discrete variables by zipping their values in a 2-tuple and applying the formula: where n is the number of distinct classes and q is the probability of ...
0
votes
0answers
28 views

minimize large time series dataset but keep its shape

I have a large time series dataset. My plan is to minimize it somehow so I can work more efficiently with this dataset while the "shape" of the data will be retained. I've read about moving averages (...
1
vote
0answers
10 views

Weights in adaboost/decision tree(cart)

I'm trying to implement adaboost using decision trees. But I'm confused over the weights. I am unable to understand how to incorporate weights in training process, how the formulas for node entropy ...
0
votes
0answers
10 views

Gini impurity and generalization error

Has anyone seen papers on relationship between information-based criterions (such as Gini impurity, information gain etc.) and generalization error? Is there theoretical justification of using such ...
1
vote
1answer
96 views

Entropy of the multinomial distribution

In my work I've found myself in the position of needing to calculate the entropy of the multinomial distribution: $$\text{Multinomial}({\bf x};\; n,{\bf p})$$ I imagine it would be too much to ...
1
vote
0answers
15 views

Looking for derivation of Shannon entropy of binomial distribution

According to Wikipedia, the Shannon entropy of a binomial distribution is $\frac{1}{2}log_2 (2\pi*e* np(1-p)) + O(1/n)$ Does anyone know where I can find a derivation?
0
votes
1answer
36 views

Determine how much order/structure is present in a 2D array of values

I have a large 2D array of numbers that may or may not be totally random. I would like to examine this array for signs of non-randomness, such as repeating patterns or other two-dimensional structures....
1
vote
1answer
59 views

Gradient of softmax with cross entropy loss

I'm working on implementing a simple deep model which uses cross-entropy loss, while using softmax to generate predictions. More specifically, I am interested in obtaining the gradient of $$CE(...
0
votes
0answers
41 views

The formal proof of purity gain (information gain) formula for decision trees

Suppose we are constructing a binary decision tree, and are using gini impurity (purity gain) to choose the best feature for splitting a node. We also have only binary features and only two classes. ...
1
vote
1answer
44 views

How is my definition of “information” different than Shannon's entropy?

I was trying to help some person here, but then I discovered that I actually almost know entropy (but not fully). Questions: Q1: how is my definition of information different than that of Claude ...
1
vote
1answer
85 views

Binning Continuous Variables By Entropy From Binary Response (R)

I am working with at data set where the goal is to predict a binary response. I have a few continuous variable that I think would be beneficial to bin. I was reading this idea about entropy based ...
0
votes
0answers
16 views

Compare two distributions with varying focus on different regions

I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ...
1
vote
1answer
94 views

What is the perplexity of a mini-language of numbers [0-9] where 0 has prob 10 times the other numbers?

I'm reading Speech and Language Processing, Jurafsky and Martin, in particular chapter 4 where they introduce perplexity see https://web.stanford.edu/~jurafsky/slp3/4.pdf (page 8-9) Here a brief ...
1
vote
1answer
141 views

Correct way of computing Shannon Entropy of a walk

Take for example a walk such as: ["school", "work", "home", "kindergarten", "home", "school", ...] # or simply [1, 2, 3, 4, 3, 1, ...] What's the correct way of ...
0
votes
1answer
46 views

An entropy and mutual information problem

Let's suppose we have 4 random variables X,Y,Z and T and that the following equations hold about the entropy: $$H(T|X)=H(T)$$ $$H(T|X,Y)=0$$ $$H(T|Y)=H(T)$$ $$H(Y|Z)=0$$ $$H(T|Z)=0$$ I want to prove ...
3
votes
0answers
25 views

Total Correlation with Renyi Entropy

The measure total correlation is defined making use of Shannon's entropy: $$ TC(X_1,\dots,X_m) = \sum_{i} H(X_i) - H(X_1,\dots,X_m) $$ This comes also with different names: e.g. multi information, or ...
0
votes
0answers
34 views

Can I increase Pointwise Mutual Information scores with a bandwidth or threshold?

I'm calculating PMI (Pointwise Mutual Information) wikipedia formula values between two time series x and y using the formula: $PMI(x, y) = \log( \frac{p(x,y)} {p(x) * p(y)} )$ where $p(x), p(y)$ ...
0
votes
0answers
20 views

Learning the Confidence of a Neural Network

Suppose I want to train a deep neural network for classification. The network takes an input vector $x$, and maps this to an output vector $y$. Now, $x$ is of length $n$ and is in fact composed of a ...
1
vote
0answers
34 views

Got an entropy-ish function for a multinomial distribution? Graph theory and Bayes net related

I have a discrete variable $X$ that can take on one of three states; $a$, $b$, and $c$. Thus it has two parameters $p_a = P(X = a)$ and $p_b = P(X = b)$, of course $P(X = c) = 1 - p_a - p_b$. I am ...
2
votes
0answers
39 views

Limit of quantized entropy

Consider the continuous random variable $X \in \mathbb R$ and the deterministic function $Y = f(X)$, $f: \mathbb R \to \mathbb R$. For the conditional entropy we have $$H(Y|X) = 0.$$ Does this imply $$...
0
votes
0answers
19 views

How to compare 2 estimations of a probability distribution?

I am using a program that returns a probability estimation Q to predict a value. I have access to the real probability distribution P that I try to estimate, from the dataset. I can compute the ...
2
votes
0answers
16 views

Difference between standard deviation and entropy [closed]

What is the difference between standard deviation and entropy in behavior of function?
0
votes
0answers
17 views

Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
0
votes
1answer
109 views

Cross entropy loss function and division by zero

I'm trying out the cross entropy loss function for neural network training, per the arguments at https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-...
26
votes
1answer
380 views

What does entropy tell us?

I am reading about entropy and am having a hard time conceptualizing what it means in the continuous case. The wiki page states the following: The probability distribution of the events, coupled ...
11
votes
2answers
275 views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
5
votes
1answer
61 views

Does Random Forest ever compare the splitting of one node to the slitting of a **different** node?

I thought I understood how a single decision tree is constructed as part of a Random Forest : The data is split recursively until some kind of stopping conditions are met. Each split is ...
2
votes
0answers
59 views

Using entropy to imputing missing value based on grey relational analysis and clustering

This algorithm contain three techniques : 1-fuzzy c-mean clustering 2-Grey relational theory 3-Entropy multiple imputation The frame work of this algorithm is as follows : My questions are ...
1
vote
0answers
24 views

Is it possible for decision trees to increase entropy of a dataset

Is it possible for decision trees to split a set of data into subsets where the sum of the entropies (weighted respectively) of each subset will be greater than the original entropy of the set? I ...
2
votes
0answers
30 views

Entropy measure for multiple node data splitting with post-normalization constraint

Let me introduce my problem with a simple example. Let's say that we have two different classes $C_0$ and $C_1$ and we have one node $S$ that has the following elements of each class: $S = \{1000,...
4
votes
1answer
79 views

Weighted entropy as a measure of diversity

Suppose that you are a company manager and you are looking for a statistical measure that defines the international reputation of your company. So, you collect data on your clients and the countries ...
1
vote
1answer
24 views

How to compute the entropy of the specific conditional probability

Consider a random variable $X$ to be uniformly distributed in a domain $\Omega$, $$P(x) = 1/|\Omega|$$, with $|\Omega| = \int_\Omega dx$. Consider a second random variable defined by $Y=f(X)$, such ...