A mathematical quantity designed to measure the amount of randomness of a random variable.

learn more… | top users | synonyms

1
vote
0answers
7 views

Calculating Diversity using normalized species counts

I am trying to calculate the change in diversity of species encountered throughout time. My dataset is comprised to detections of tagged fish for 48 consecutive weeks. My thought was to bin the ...
0
votes
0answers
26 views

Entropy of multivariate gaussian mixture random variable

Short: ${\bf X} \sim N({\bf 0},{\bf I}+{\bf I}_j)$; ${\bf I}_j\in S=\{I_j: I_j$ is diagonal and $ I_j \succeq 0\}, |S|=K$, and $j\sim U(1,K)$. What is $h({\bf X})$? What happens when ...
0
votes
0answers
10 views

Conditional Entropy and spearman Correlation based lag in time series

I have two time series A, B. Both are seasonal and B primarily is A driven( other temporal causes may occur). B-Red, A- Green I want to calculate lag of red series with respect to green as clearly, ...
0
votes
0answers
6 views

Transfer entropy on real, continuous data

Here is the goal and problem: I am trying to calculate a measure of coupling between real-valued, continuous oscillatory data. The data come from two people producing synchronized rhythmic ...
0
votes
0answers
11 views

Residuals as a percentage

Considering that still there is no suggestions, I assumed that my initial question is not fully understandable. I will try to ask same question in more abstract way, hoping it will come out as better ...
0
votes
1answer
23 views

Find entropy in WEKA

I am new in data mining so sorry for asking this kind of silly question. I am working on FAST feature selection algorithm and for that I need to find entropy of each attribute in dataset. But the ...
0
votes
0answers
9 views

R “Entropy” package gives weird KL divergence results

Using the R "entropy" package, I tried some KL divergence computations as a sanity check, but I'm getting weird results. For instance, shouldn't the following all be 2*log2(2)= 2 ? Instead, I'm ...
1
vote
1answer
16 views

How does the log(p(x,y)) normalize the pointwise mutual information?

I'm trying to understand the normalized form of pointwise mutual information. $npmi = \frac{pmi(x,y)}{log(p(x,y))}$ Why does the log joint probability normalize the pointwise mutual information to ...
0
votes
0answers
23 views

Entropy, Softmax and the derivative term in Backpropagation

I'm currently interested in using Cross Entropy Error when performing the BackPropagation algorithm for classification, where I use the Softmax Activation Function in my output layer. From what I ...
1
vote
0answers
22 views

Maximum Entropy with no index

This is a simpler problem than trying to solve, but have a feeling once get the methodology I can apply it to the harder problem. Let $ H(p)= -q \ln(q) - p \ln(p) $ be the entropy of the Bernoulli ...
0
votes
1answer
51 views

Sum of truncated normal distributions with other distributions (like uniform) that have partially common domains

I have some truncated normal distributions and some other distributions (like uniform distribution) that have partially common domains. I'd like to know how can I calculate the entropy for the sum of ...
1
vote
1answer
43 views

Tsallis and Renyi Normalized Entropy

I'm working with Shannon, Tsallis and Renyi entropies. I need to normalized these entropies for comparison purposes. In Shannon's entropy you need only to divide by the log of the number of bins. ...
1
vote
0answers
23 views

Propagation of uncertainty: entropy of multinomial

My goal is to estimate the entropy of a multinomial distribution, based on a single observation (a set of counts for each possible outcome). I also want to calculate the uncertainty in my estimated ...
1
vote
0answers
50 views

Decision tree with adaboost

Helllo! I'm currently learning the AdaBoost algorithm to use it with Decision Tree. I want to implement everything myself (that's the way I learn - implement everything from scratch and later use ...
4
votes
2answers
82 views

Is it possible to use SD instead of entropy?

While discussing about decision trees in class, my teacher touched upon the topic of entropy. I have understood the purpose of entropy (have not understood how the formula $H(X)= -\sum_{i}{p(x_i) \log ...
1
vote
1answer
33 views

Why to use Chi2 instead of accuracy in a decision tree?

Why many decision trees are using Chi2 or Information Gain Ratio to split the node when they can directly use accuracy, lift or AUC?
3
votes
0answers
59 views

Computation of the entropy of marginals?

I have implemented this paper: Efficient graph-based semi-supervised learning of structured tagging models In the last sentence of the section 4.2, the authors have mentioned another possible way of ...
5
votes
1answer
115 views

How to determine Forecastability of time series?

One of the important issues facing forecasters is if the given series can be forecasted or not ? I stumbled on an article entitled "Entropy as an A Priori Indicator of Forecastability" by Peter ...
3
votes
1answer
58 views

Entropy of Sum vs Difference of Random Variable

I am looking for a proof of the following fact Let $X$ and$ X'$ be i.i.d on $\{0,1,2\}$ (not necessarily uniform). Prove that $$H((X + X') \mod3) \leq H((X - X') \mod3)$$ where $H()$ is the ...
0
votes
1answer
98 views

Is there an R package to calculate differential entropy

I am trying to calculate differential entropy over my data. This is how a subset of my data set looks like : ...
0
votes
0answers
26 views

structural equation model based on generalized maximum entropy in R

I am trying to conduct an experiment based on generalized maximum entropy but I am not sure on how GME is different from maximum entropy. Can anybody tell me how to reparametized the SEM based on GME ...
2
votes
2answers
92 views

Why KL-Divergence uses “ln” in its formula?

I notice in KL-Divergence formula a $ln$ function is used: $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability ...
2
votes
1answer
72 views

Entropy stays the same with a larger distribution?

Based on the given definition of entropy, $H(P(X)) = -\sum_i P(x_i)log_2(P(x_i)$, it appears that if I have a distribution $P_1(x) = [\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{4}]$ and another ...
0
votes
0answers
40 views

What is the relationship between Shannon Entropy and Approximate Entropy

I prefer a worded explanation because I am not a math expert. I know it Approximate Entropy is derived as an aproximation of Kolmogorov Entropry so the relationship between Kolmogorov Entropry and ...
1
vote
1answer
63 views

Why constrain mean and standard deviation when proving Gaussian is maximum differential entropy pdf?

I'm reading Bishop's Pattern Recognition and Machine Learning. In chapter 1.6: Information Theory (page 53) when trying to derive the maximum differential entropy pdf from the definition of continuous ...
0
votes
0answers
77 views

Difference between Weighted Average Entropy and Adjusted Mutual Information (for evaluating Clustering)

I was advised by my team leader to use this weighted average entropy to evaluating the performance of my dbscan clustering algorithm, and its mathematical formulation is: Scikit provides what many ...
2
votes
0answers
24 views

Continuous variable evaluation in decision trees

I was going through the C4.5 and ID3 algorithms used to construct a decision tree. Was wondering if there is an efficient way to compute information gain from a continuous variable (during the step ...
0
votes
2answers
33 views

Entropy of a block of characters

I have a question about the following statement about entropy: If a source provides us with a sequence chosen from 4 symbols (say A, C, G, T), then the maximum average information per symbol is 2 ...
2
votes
0answers
136 views

How to split a decision tree when information gains of all attributes are zero?

The textbook tells us that we should choose an attribute with the maximum information gain to split a decision tree. My question is what if all information gains are zero? Should we stop splitting or ...
0
votes
0answers
12 views

Model comparision

I am comparing supervised vs unsupervised method. I have computed entropy from the confusion matrix of the 10-fold crossvalidated output. I have also applied external criteria to unsupervised methods ...
0
votes
0answers
58 views

Constructing Random Forests for binary classification by minimizing entropy

I'm looking to perform a binary classification using random forests, but I do not quite understand how to minimize the entropy of the data / what tests I should run on the nodes to do so. I'm fairly ...
1
vote
0answers
61 views

Likelihood Ratio as statistical test for Transfer Entropy with MatLab

I estimated a Transfer Entropy value TE(XY). Now I want to establish the statistical significance of the estimated value. Therefore I used the method of shuffled surrogates to estimate a shuffled ...
1
vote
1answer
39 views

Significance test for entropy?

Is there any way to test the difference of entropy given frequency table? For example, let's say we have dice 1 and dice 2, and we experimented with them and they showed up like ...
0
votes
1answer
69 views

Why is this a good approximation of cross entropy?

Given a random variable $X=x_i$ for $i=1,...,n$, with true distribution $p(x)$ and approximate distribution $q(x)$, its cross entropy is given by $$H(p,q) = -\sum_{i=1}^np(x_i)\log q(x_i).$$ However ...
1
vote
1answer
29 views

Simple measure for comparison of rainfall regularity

In meteorology we have the concept of monthly rainfall, which is just the sum of daily rainfall over that month. Now, given this extreme example: Situation 1: ...
0
votes
0answers
25 views

Entropy weighted Naive Bayes performs poorer than regular Naive Bayes?

I have a text classification problem, where there are many different classes, and the text to be classified is very short (about 1 sentence each): ...
3
votes
1answer
100 views

Compare skewness of many distributions with few observations

I have a dataset with page view data for about 500,000 users, divided into two groups. Each user can visit up to 5 pages, each as many or as few times as they want. So for each user, I have the ...
1
vote
1answer
89 views

Conceptual question on mutual information and entropy

What does mutual information (MI) convey? Looking for good reference books on information theory
1
vote
1answer
47 views

Conceptual questions on Entropy and estimation

Learning Informative Statistics: A Nonparametric Approach paper presents an approach to parameter estimation by entropy minimization. There are other related works "Minimum-entropy estimation in ...
0
votes
1answer
56 views

Do information entropy probabilities have to sum to one?

My understanding of information entropy is that it requires the input probabilities to sum to 1. So, for a sequence a,a,b,b you then have $$- \left(\frac12 \log_2 \frac12 + \frac12 \log_2 ...
1
vote
1answer
56 views

Differentiating Shannon's entropy [closed]

Can somebody please show the steps of how differentiation of Shannon's entropy yields the following result? $H = -\sum_{l=0}^{L-1} p(l)\log_2[p(l)]$ The result of differentiating is $H_m = ...
1
vote
0answers
68 views

What does Class Complexity mean in Weka?

When running Weka on my dataset, in the results printout I get the following rows: ...
1
vote
0answers
54 views

Neuroscience Equations

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and ...
0
votes
0answers
20 views

General reference to Rényi entropy

Is there a book or a review article serving as a good general reference to Rényi entropy, its applications and related concepts?
1
vote
1answer
147 views

Calculation of relative information gain for multiple splits in decision trees

As it well known, we can calculate Relative Information gain (RIG) as follows: $RIG = \frac{H(x) - H(x|a)}{H(x)}$. In binary decision trees we calculate $H(x|a)$ for univariate split for variable ...
1
vote
1answer
1k views

Feature selection : how to select the Information Gain threshold?

I am trying to use Information Gain to select features when classifying text with a Support Vector Machine. For each word in our training data, we computed its information gain. Then, we should keep ...
0
votes
1answer
47 views

Differential Entropy of Gaussian Process

I have $N$ datapoints that have $d$ features in a GP and their covariance matrix $K$ and I want to calculate the differential entropy of that GP. Is this formula right? $E(I)= \frac{1}{2} ...
0
votes
1answer
323 views

Information gain with numerical data

I'm making a random forest classifier. In every tutorial, there is a very simple example of how to calculate entropy with Boolean attributes. In my problem I have attribute values that are calculated ...
0
votes
1answer
151 views

Relationship between entropy and information gain

Based on papers :1. Deniz Erdogmus, Member, IEEE, and Jose C. Principe, An Error-Entropy Minimization Algorithm for Supervised Training of Nonlinear Adaptive Systems J. Principe, D. Xu, and J. ...
0
votes
1answer
83 views

Entropy and information content

I am curious to know about the relation between Entropy and information content of a signal or trajectory of time series. When a system is at equilibrium, then it has maximum entropy. Does entropy ...