A mathematical quantity designed to measure the amount of randomness of a random variable.

learn more… | top users | synonyms

2
votes
0answers
12 views

Difference between standard deviation and entropy [on hold]

What is the difference between standard deviation and entropy in behavior of function?
0
votes
0answers
6 views

Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
0
votes
1answer
12 views

Cross entropy loss function and division by zero

I'm trying out the cross entropy loss function for neural network training, per the arguments at ...
19
votes
1answer
225 views
+50

What does entropy tell us?

I am reading about entropy and am having a hard time conceptualizing what it means in the continuous case. The wiki page states the following: The probability distribution of the events, coupled ...
9
votes
2answers
222 views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
5
votes
1answer
48 views

Does Random Forest ever compare the splitting of one node to the slitting of a **different** node?

I thought I understood how a single decision tree is constructed as part of a Random Forest : The data is split recursively until some kind of stopping conditions are met. Each split is ...
1
vote
0answers
39 views

Using entropy to imputing missing value based on grey relational analysis and clustering

This algorithm contain three techniques : 1-fuzzy c-mean clustering 2-Grey relational theory 3-Entropy multiple imputation The frame work of this algorithm is as follows : My questions are ...
1
vote
0answers
18 views

Is it possible for decision trees to increase entropy of a dataset

Is it possible for decision trees to split a set of data into subsets where the sum of the entropies (weighted respectively) of each subset will be greater than the original entropy of the set? I ...
2
votes
0answers
18 views

Entropy measure for multiple node data splitting with post-normalization constraint

Let me introduce my problem with a simple example. Let's say that we have two different classes $C_0$ and $C_1$ and we have one node $S$ that has the following elements of each class: $S = ...
4
votes
1answer
53 views

Weighted entropy as a measure of diversity

Suppose that you are a company manager and you are looking for a statistical measure that defines the international reputation of your company. So, you collect data on your clients and the countries ...
1
vote
1answer
22 views

How to compute the entropy of the specific conditional probability

Consider a random variable $X$ to be uniformly distributed in a domain $\Omega$, $$P(x) = 1/|\Omega|$$, with $|\Omega| = \int_\Omega dx$. Consider a second random variable defined by $Y=f(X)$, such ...
0
votes
0answers
16 views

Relative information gain (RIG) calculation for scored binary classifier

I'm interested in calculating RIG from scoring binary classifier output (i.e. two column data set: prediction/score, labels). I'm planning to use it for feature selection and estimating improvements. ...
0
votes
0answers
13 views

Zero-error vs non-zero-error Shannon capacity

I'm trying to solve the following problem on channel capacities, but for some reason I seem to go in circles. Let $P_{Y|X}$ be a discrete memoryless channel with confusability graph $G$ and ...
0
votes
1answer
44 views

Entropy of a Random Variable

So I know how to calculate entropy given a specific probability: E.g. H(0.5, 0.5) = 1 bit However, I'm a bit unsure how to handle the random variable case. For instance: Let X, Y be some jointly ...
3
votes
1answer
83 views

maximum value of $H(X_1+X_2)$

Let $X_1$ & $X_2$ be two independent binary r.v.s, taking values in {0,1}; let $Y \triangleq X_1+X_2$ be their sum, i.e. a ternary r.v. taking values in {0,1,2}. How do we prove that the maximum ...
2
votes
0answers
37 views

Is it possible to get a negative infinite differential entropy without delta function and limit?

If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?
2
votes
0answers
43 views

Maximum Entropy without the probabilities

The entropy of a distribution is defined as $H = -\sum_{i=1}^n p_i \log(p_1)$ The principle of maximum entropy states that we should choose the distribution, subject to our constraints, that ...
13
votes
1answer
272 views

What is the meaning of the eigenvectors of a mutual information matrix?

When looking at the eigenvectors of the covariance matrix, we get the directions of maximum variance (the first eigenvector is the direction in which the data varies the most, etc.); this is called ...
2
votes
0answers
16 views

Confusion in calculation of conditional entropy

I'm following a course on Information Theory and came across a problem about conditional entropy that I thought I understood, but when I tried calculating the actual values my results didn't satisfy ...
4
votes
0answers
63 views

How to estimate a probability distribution

Suppose I want to estimate a probability distribution, is it common practice to simply fit a function to a frequency histogram? So in my work, I am training a classifier, the performance of which is ...
2
votes
1answer
87 views

Additivity of Shannon's entropy

E.T. Jaynes writes, in "Probability Theory: the Logic of Science" the following in order to motivate the derivation of the entropy, $H$: Suppose the robot perceives two alternatives, to which it ...
0
votes
0answers
14 views

information entropy of given sequence

consider I have Loaded coin of P(H) = 0.6 and P(T) = 0.4 there is a sequence of coin X = HHHTT then how do I calculate information entropy of X?? is it (-(6/10)*(log(6/10))) * 3 + ...
0
votes
0answers
11 views

Decision tree - choosing which attribute gives a better prediction for a different attribute

I did one exercise as HW a while ago and I am not sure if whether is right or wrong. I would like your opinion and if it is wrong what would be the correct way to do it. The problem was the ...
1
vote
0answers
24 views

Entropy rate for continuous variables?

As far as I know, the entropy rate of a random process is defined as $$ h = \lim_{n \rightarrow \infty} \frac{1}{n}H(X_1, ..., H_n) $$ In conventional, finite-alphabet information theory the entropy ...
1
vote
0answers
18 views

Clarification on statistical notation of mutual information

The following is a definition from a textbook about information theory: Definition: The conditional mutual information of random varaibles X and Y given Z is defined by $I(X;Y|Z) = ...
0
votes
0answers
17 views

Comparing two features capturing the same aspect but calculated differently?

In a classification problem I am trying to solve, one of the input (independent) features can be calculated by using two alternative calculation methods. One way of comparing the two methods is to see ...
2
votes
0answers
49 views

joint differential entropy $h(X,Y)$, when $Y=g(X)$

It's well known that if X & Y are discrete random variables X & Y (r.v.s), and $Y=g(X)$, then $$H(X,Y)=H(X)+H(Y\mid X)=H(X),$$ where the last equality is due to $H(Y\mid X)=0.$ It also has ...
4
votes
1answer
55 views

If a decision tree already has very low entropy, do we still need a random forest

I need some help understanding the concept of random forests. As I understand, when I make a decision tree, I carefully select each node so as to maximize the information gain and minimize the ...
1
vote
0answers
59 views

Name of an $f$-divergence

The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$. The relative ...
2
votes
1answer
137 views

Relationship between least-squares regression and information theory

Is there a well-known relationship between least-squares regression and information theory? I've just started reading about information theory. It seems almost trivial to say that the regression ...
1
vote
0answers
572 views

Multinomial Logistic Loss vs (Cross Entropy vs Square Error)

I observed that Caffe (a deep learning framework) used the Softmax Loss Layer SoftmaxWithLoss as output layer for most of the model samples. As far as I know, ...
2
votes
1answer
300 views

How to apply Cross Entropy on Rectified Linear Units

I am currently getting started with Machine Learning. However, I have some problem to derive formula and not able understand how to applied the Cross Entropy (CE) on Rectified Linear Units (ReLU). I ...
4
votes
1answer
68 views

Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
0
votes
0answers
14 views

Generalized mutual information fulfilling data processing inequality

One of key features of mutual information $I(X,Y)$ is that it cannot be increased by local operations. That is, if we perform any local operations $X \to X'$ or $Y \to Y'$ then $$I(X', Y') \leq I(X, ...
0
votes
1answer
36 views

When is the conditional differential entropy, $h(X+Z_1\mid X+Z_2)$, maximized?

Let $Z_1$ & $Z_2$ be 2 i.i.d. RVs, each distributed according to $N(0,1)$, and let $X$ be an arbitrary RV with unit variance. What distribution of X will maximize this conditional differential ...
3
votes
1answer
61 views

Does correlation implies mutual information?

I am skeptical about the notion that if mutual information between two random variables is non-zero (existence) this doesn't imply the existence of correlation between them BUT existence of ...
1
vote
1answer
37 views

Relative information gain for continuous RV

I am looking for a well-defined measure of the information gained about a vector $X$ from observing another vector $Y$. Both $X$ and $Y$ are continuous random vectors. For example, suppose that ...
0
votes
0answers
38 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
4
votes
1answer
80 views

Sum of squared Negative Binomial probability masses

Let $(p_k)_{k=0, \dots, \infty}$ denote the probability masses of a Negative Binomial distribution with parameters $r>0$ and $p\in]0,1[$. I'm looking for the sum of their squares, ...
5
votes
1answer
191 views

Sum of squared Poisson probability masses

Let $(p_k)_{k=0, \dots, \infty}$ denote the probability masses of a Poisson distribution with parameter $\lambda$. I'm looking for the sum of their squares, $$\sum_{k=0}^\infty p_k^2,$$ as a function ...
0
votes
1answer
135 views

How to find a Information Gain for Numerical Values

I have 4 class and four sets of features extracted from the huge data extracted from Real Time data acquisition system. . In above table, there are 4 sets of the features for each class. (C1, C2, C3 ...
4
votes
2answers
501 views

Why would perfectly similar data have 0 mutual information?

I'm not a statistic major, so my knowledge of statistics is quite limited but I've found myself in need of learning about and using mutual information. I believe I understand the concept and formula, ...
0
votes
0answers
38 views

How is the entropy calculated by the `entropy` package in R?

So, as per the docs, I'm calling the function like this v = c(0,4,3,6,7,3,2,3,4,5) entropy(discretize(v, numBins = 8, r = c(0,7))) and I get ...
2
votes
0answers
18 views

Self-designed objective for multiple linear regression

A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship: ...
1
vote
1answer
44 views

How an entropy can be used to cluster a stocks in stock market? [closed]

Recently when I read reviews in entropy and its application in share market. I found the concept of entropy and clustering is not actively used in stock market analysis. I think the clustering stocks ...
0
votes
1answer
22 views

Why do we not care about target class probabilities of 0 in the cross entropy equation?

Only the target classes where probability is equal to 1 contributes to the loss. I'm dealing specifically with neural networks, but the question is general.
9
votes
1answer
194 views

Is differential entropy always less than infinity?

For an arbitrary continuous random variable, say $X$, is its differential entropy always less than $\infty$? (It's ok if it's $-\infty$.) If not, what's the necessary and sufficient condition for it ...
2
votes
0answers
30 views

Impact of conjugate priors on mutual information for Naive Bayes

I am currently thinking about the following problem. Suppose you have a simple Naive Bayes model for binary classification based on binary random variables. For example, suppose you want to predict ...
2
votes
1answer
57 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove that $$ H(X) \leq H(Y) \implies ...
0
votes
0answers
20 views

Minimum description length principal (MDLP) stopping criterion with repeated values

Fayyad's paper on discretization of continuous attributes describes a canonical way to discretize continuous attributes using a minimum entropy method. However, the paper does not describe what to do ...