A mathematical quantity designed to measure the amount of randomness of a random variable.

learn more… | top users | synonyms

3
votes
0answers
21 views

Gibbs entropy and Shannon entropy

The two formulations seem identical to me: $H(x) = \sum p(x) log(1/p(x))$ why would the equation be attributed to Shannon rather than Gibbs (in the context of information)?
1
vote
0answers
4 views

Calculating joint entropy of two variables having null values

I calculated the joint entropy of two random discrete variables by zipping their values in a 2-tuple and applying the formula: where n is the number of distinct classes and q is the probability of ...
0
votes
0answers
26 views

minimize large time series dataset but keep its shape

I have a large time series dataset. My plan is to minimize it somehow so I can work more efficiently with this dataset while the "shape" of the data will be retained. I've read about moving averages ...
0
votes
0answers
3 views

Weights in adaboost/decision tree(cart)

I'm trying to implement adaboost using decision trees. But I'm confused over the weights. I am unable to understand how to incorporate weights in training process, how the formulas for node entropy ...
0
votes
0answers
6 views

Gini impurity and generalization error

Has anyone seen papers on relationship between information-based criterions (such as Gini impurity, information gain etc.) and generalization error? Is there theoretical justification of using such ...
1
vote
1answer
55 views

Entropy of the multinomial distribution

In my work I've found myself in the position of needing to calculate the entropy of the multinomial distribution: $$\text{Multinomial}({\bf x};\; n,{\bf p})$$ I imagine it would be too much to ...
1
vote
0answers
12 views

Looking for derivation of Shannon entropy of binomial distribution

According to Wikipedia, the Shannon entropy of a binomial distribution is $\frac{1}{2}log_2 (2\pi*e* np(1-p)) + O(1/n)$ Does anyone know where I can find a derivation?
0
votes
1answer
32 views

Determine how much order/structure is present in a 2D array of values

I have a large 2D array of numbers that may or may not be totally random. I would like to examine this array for signs of non-randomness, such as repeating patterns or other two-dimensional ...
1
vote
1answer
32 views

Gradient of softmax with cross entropy loss

I'm working on implementing a simple deep model which uses cross-entropy loss, while using softmax to generate predictions. More specifically, I am interested in obtaining the gradient of ...
0
votes
0answers
25 views

The formal proof of purity gain (information gain) formula for decision trees

Suppose we are constructing a binary decision tree, and are using gini impurity (purity gain) to choose the best feature for splitting a node. We also have only binary features and only two classes. ...
1
vote
1answer
42 views

How is my definition of “information” different than Shannon's entropy?

I was trying to help some person here, but then I discovered that I actually almost know entropy (but not fully). Questions: Q1: how is my definition of information different than that of Claude ...
1
vote
1answer
40 views

Binning Continuous Variables By Entropy From Binary Response (R)

I am working with at data set where the goal is to predict a binary response. I have a few continuous variable that I think would be beneficial to bin. I was reading this idea about entropy based ...
0
votes
0answers
14 views

Compare two distributions with varying focus on different regions

I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ...
1
vote
1answer
90 views

What is the perplexity of a mini-language of numbers [0-9] where 0 has prob 10 times the other numbers?

I'm reading Speech and Language Processing, Jurafsky and Martin, in particular chapter 4 where they introduce perplexity see https://web.stanford.edu/~jurafsky/slp3/4.pdf (page 8-9) Here a brief ...
1
vote
0answers
47 views

Correct way of computing Shannon Entropy of a walk

Take for example a walk such as: ["school", "work", "home", "kindergarten", "home", "school", ...] # or simply [1, 2, 3, 4, 3, 1, ...] What's the correct way of ...
0
votes
1answer
44 views

An entropy and mutual information problem

Let's suppose we have 4 random variables X,Y,Z and T and that the following equations hold about the entropy: $$H(T|X)=H(T)$$ $$H(T|X,Y)=0$$ $$H(T|Y)=H(T)$$ $$H(Y|Z)=0$$ $$H(T|Z)=0$$ I want to prove ...
3
votes
0answers
19 views

Total Correlation with Renyi Entropy

The measure total correlation is defined making use of Shannon's entropy: $$ TC(X_1,\dots,X_m) = \sum_{i} H(X_i) - H(X_1,\dots,X_m) $$ This comes also with different names: e.g. multi information, or ...
0
votes
0answers
31 views

Can I increase Pointwise Mutual Information scores with a bandwidth or threshold?

I'm calculating PMI (Pointwise Mutual Information) wikipedia formula values between two time series x and y using the formula: $PMI(x, y) = \log( \frac{p(x,y)} {p(x) * p(y)} )$ where $p(x), p(y)$ ...
0
votes
0answers
16 views

Learning the Confidence of a Neural Network

Suppose I want to train a deep neural network for classification. The network takes an input vector $x$, and maps this to an output vector $y$. Now, $x$ is of length $n$ and is in fact composed of a ...
0
votes
0answers
17 views

Got an entropy-ish function for a multinomial distribution? Graph theory and Bayes net related

I have a discrete variable $X$ that can take on one of three states; $a$, $b$, and $c$. Thus it has two parameters $p_a = P(X = a)$ and $p_b = P(X = b)$, of course $P(X = c) = 1 - p_a - p_b$. I am ...
2
votes
0answers
39 views

Limit of quantized entropy

Consider the continuous random variable $X \in \mathbb R$ and the deterministic function $Y = f(X)$, $f: \mathbb R \to \mathbb R$. For the conditional entropy we have $$H(Y|X) = 0.$$ Does this imply ...
0
votes
0answers
18 views

How to compare 2 estimations of a probability distribution?

I am using a program that returns a probability estimation Q to predict a value. I have access to the real probability distribution P that I try to estimate, from the dataset. I can compute the ...
2
votes
0answers
16 views

Difference between standard deviation and entropy [closed]

What is the difference between standard deviation and entropy in behavior of function?
0
votes
0answers
13 views

Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
0
votes
1answer
64 views

Cross entropy loss function and division by zero

I'm trying out the cross entropy loss function for neural network training, per the arguments at ...
24
votes
1answer
339 views

What does entropy tell us?

I am reading about entropy and am having a hard time conceptualizing what it means in the continuous case. The wiki page states the following: The probability distribution of the events, coupled ...
9
votes
2answers
260 views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
5
votes
1answer
56 views

Does Random Forest ever compare the splitting of one node to the slitting of a **different** node?

I thought I understood how a single decision tree is constructed as part of a Random Forest : The data is split recursively until some kind of stopping conditions are met. Each split is ...
1
vote
0answers
51 views

Using entropy to imputing missing value based on grey relational analysis and clustering

This algorithm contain three techniques : 1-fuzzy c-mean clustering 2-Grey relational theory 3-Entropy multiple imputation The frame work of this algorithm is as follows : My questions are ...
1
vote
0answers
21 views

Is it possible for decision trees to increase entropy of a dataset

Is it possible for decision trees to split a set of data into subsets where the sum of the entropies (weighted respectively) of each subset will be greater than the original entropy of the set? I ...
2
votes
0answers
28 views

Entropy measure for multiple node data splitting with post-normalization constraint

Let me introduce my problem with a simple example. Let's say that we have two different classes $C_0$ and $C_1$ and we have one node $S$ that has the following elements of each class: $S = ...
4
votes
1answer
75 views

Weighted entropy as a measure of diversity

Suppose that you are a company manager and you are looking for a statistical measure that defines the international reputation of your company. So, you collect data on your clients and the countries ...
1
vote
1answer
23 views

How to compute the entropy of the specific conditional probability

Consider a random variable $X$ to be uniformly distributed in a domain $\Omega$, $$P(x) = 1/|\Omega|$$, with $|\Omega| = \int_\Omega dx$. Consider a second random variable defined by $Y=f(X)$, such ...
0
votes
0answers
30 views

Relative information gain (RIG) calculation for scored binary classifier

I'm interested in calculating RIG from scoring binary classifier output (i.e. two column data set: prediction/score, labels). I'm planning to use it for feature selection and estimating improvements. ...
0
votes
0answers
15 views

Zero-error vs non-zero-error Shannon capacity

I'm trying to solve the following problem on channel capacities, but for some reason I seem to go in circles. Let $P_{Y|X}$ be a discrete memoryless channel with confusability graph $G$ and ...
0
votes
1answer
52 views

Entropy of a Random Variable

So I know how to calculate entropy given a specific probability: E.g. H(0.5, 0.5) = 1 bit However, I'm a bit unsure how to handle the random variable case. For instance: Let X, Y be some jointly ...
3
votes
1answer
86 views

maximum value of $H(X_1+X_2)$

Let $X_1$ & $X_2$ be two independent binary r.v.s, taking values in {0,1}; let $Y \triangleq X_1+X_2$ be their sum, i.e. a ternary r.v. taking values in {0,1,2}. How do we prove that the maximum ...
3
votes
0answers
43 views

Is it possible to get a negative infinite differential entropy without delta function and limit?

If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?
2
votes
0answers
48 views

Maximum Entropy without the probabilities

The entropy of a distribution is defined as $H = -\sum_{i=1}^n p_i \log(p_1)$ The principle of maximum entropy states that we should choose the distribution, subject to our constraints, that ...
13
votes
1answer
308 views

What is the meaning of the eigenvectors of a mutual information matrix?

When looking at the eigenvectors of the covariance matrix, we get the directions of maximum variance (the first eigenvector is the direction in which the data varies the most, etc.); this is called ...
2
votes
0answers
19 views

Confusion in calculation of conditional entropy

I'm following a course on Information Theory and came across a problem about conditional entropy that I thought I understood, but when I tried calculating the actual values my results didn't satisfy ...
4
votes
0answers
69 views

How to estimate a probability distribution

Suppose I want to estimate a probability distribution, is it common practice to simply fit a function to a frequency histogram? So in my work, I am training a classifier, the performance of which is ...
2
votes
1answer
114 views

Additivity of Shannon's entropy

E.T. Jaynes writes, in "Probability Theory: the Logic of Science" the following in order to motivate the derivation of the entropy, $H$: Suppose the robot perceives two alternatives, to which it ...
0
votes
0answers
17 views

information entropy of given sequence

consider I have Loaded coin of P(H) = 0.6 and P(T) = 0.4 there is a sequence of coin X = HHHTT then how do I calculate information entropy of X?? is it (-(6/10)*(log(6/10))) * 3 + ...
0
votes
0answers
14 views

Decision tree - choosing which attribute gives a better prediction for a different attribute

I did one exercise as HW a while ago and I am not sure if whether is right or wrong. I would like your opinion and if it is wrong what would be the correct way to do it. The problem was the ...
1
vote
0answers
30 views

Entropy rate for continuous variables?

As far as I know, the entropy rate of a random process is defined as $$ h = \lim_{n \rightarrow \infty} \frac{1}{n}H(X_1, ..., H_n) $$ In conventional, finite-alphabet information theory the entropy ...
1
vote
0answers
21 views

Clarification on statistical notation of mutual information

The following is a definition from a textbook about information theory: Definition: The conditional mutual information of random varaibles X and Y given Z is defined by $I(X;Y|Z) = ...
0
votes
0answers
17 views

Comparing two features capturing the same aspect but calculated differently?

In a classification problem I am trying to solve, one of the input (independent) features can be calculated by using two alternative calculation methods. One way of comparing the two methods is to see ...
2
votes
0answers
58 views

joint differential entropy $h(X,Y)$, when $Y=g(X)$

It's well known that if X & Y are discrete random variables X & Y (r.v.s), and $Y=g(X)$, then $$H(X,Y)=H(X)+H(Y\mid X)=H(X),$$ where the last equality is due to $H(Y\mid X)=0.$ It also has ...
4
votes
1answer
63 views

If a decision tree already has very low entropy, do we still need a random forest

I need some help understanding the concept of random forests. As I understand, when I make a decision tree, I carefully select each node so as to maximize the information gain and minimize the ...