A mathematical quantity designed to measure the amount of randomness of a random variable.

learn more… | top users | synonyms

0
votes
0answers
10 views

When is the conditional differential entropy maximized?

Let $Z_1$ & $Z_2$ be 2 i.i.d. RVs, each distributed according to $N(0,1)$, and let $X$ be an arbitrary RV with zero mean and unit variance. What distribution of X will maximize this conditional ...
1
vote
0answers
20 views

Does correlation implies mutual information?

I am skeptical about the notion that if mutual information between two random variables is non-zero (existence) this doesn't imply the existence of correlation between them BUT existence of ...
0
votes
0answers
22 views

Relative information gain for continuous RV

I am looking for a well-defined measure of the information gained about a vector $X$ from observing another vector $Y$. Both $X$ and $Y$ are continuous random vectors. For example, suppose that ...
0
votes
0answers
12 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
4
votes
1answer
53 views

Sum of squared Negative Binomial probability masses

Let $(p_k)_{k=0, \dots, \infty}$ denote the probability masses of a Negative Binomial distribution with parameters $r>0$ and $p\in]0,1[$. I'm looking for the sum of their squares, ...
5
votes
1answer
156 views

Sum of squared Poisson probability masses

Let $(p_k)_{k=0, \dots, \infty}$ denote the probability masses of a Poisson distribution with parameter $\lambda$. I'm looking for the sum of their squares, $$\sum_{k=0}^\infty p_k^2,$$ as a function ...
0
votes
1answer
26 views

How to find a Information Gain for Numerical Values

I have 4 class and four sets of features extracted from the huge data extracted from Real Time data acquisition system. . In above table, there are 4 sets of the features for each class. (C1, C2, C3 ...
4
votes
2answers
352 views

Why would perfectly similar data have 0 mutual information?

I'm not a statistic major, so my knowledge of statistics is quite limited but I've found myself in need of learning about and using mutual information. I believe I understand the concept and formula, ...
0
votes
0answers
28 views

How is the entropy calculated by the `entropy` package in R?

So, as per the docs, I'm calling the function like this v = c(0,4,3,6,7,3,2,3,4,5) entropy(discretize(v, numBins = 8, r = c(0,7))) and I get ...
2
votes
0answers
16 views

Self-designed objective for multiple linear regression

A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship: ...
1
vote
1answer
20 views

How an entropy can be used to cluster a stocks in stock market? [closed]

Recently when I read reviews in entropy and its application in share market. I found the concept of entropy and clustering is not actively used in stock market analysis. I think the clustering stocks ...
0
votes
1answer
19 views

Why do we not care about target class probabilities of 0 in the cross entropy equation?

Only the target classes where probability is equal to 1 contributes to the loss. I'm dealing specifically with neural networks, but the question is general.
7
votes
1answer
96 views

Is differential entropy always less than infinity?

For an arbitrary continuous random variable, say $X$, is its differential entropy always less than $\infty$? (It's ok if it's $-\infty$.) If not, what's the necessary and sufficient condition for it ...
2
votes
0answers
17 views

Impact of conjugate priors on mutual information for Naive Bayes

I am currently thinking about the following problem. Suppose you have a simple Naive Bayes model for binary classification based on binary random variables. For example, suppose you want to predict ...
2
votes
1answer
44 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove that $$ H(X) \leq H(Y) \implies ...
0
votes
0answers
8 views

Minimum description length principal (MDLP) stopping criterion with repeated values

Fayyad's paper on discretization of continuous attributes describes a canonical way to discretize continuous attributes using a minimum entropy method. However, the paper does not describe what to do ...
1
vote
0answers
14 views

lower bound on conditional differential entropy

I was wondering if there exists a lower bound of conditional differential entropy of Gaussian random variables. Formally, let $\mathcal{X}=\{X_1,\ldots,X_m\}$ and $\mathcal{Y}=\{Y_1,\ldots,Y_n\}$ ...
3
votes
1answer
130 views

What is the intuitive (geometric?) meaning of minimizing the log determinant of a matrix?

I have come across optimization problems which seek a positive semi-definite matrix $A$ that minimizes some possibly non-convex function that includes the addition of $1/(\text{dimension}) * \log \det ...
2
votes
1answer
21 views

Information content of a set of random variables

Suppose there is some distribution $F$ not known to us. However, we can get information about this distribution by means of samples, i.e. we have a set of random variables from this distribution. ...
0
votes
1answer
33 views

Regularizing soft kmeans with entropy

So in classical fuzzy k-means clustering, the objective function is $\sum_i \sum_j u_{ij} \|x_i - c_j\|^2$ Now, we want to regularize this objective function using the entropy: $\sum_i^n H(U_i) = - ...
1
vote
0answers
20 views

entropy-calculation in R [closed]

Is there anyone to have used the package "entropy" ? How can I create a vector of counts if my sample is large (2000 points)? I tried creating a vector of counts using table(), is there a more ...
3
votes
2answers
68 views

software library to compute KL divergence?

Are there any software libraries that compute KL divergences in closed form, that also give the derivatives of the KL divergence wrt the distributions' parameters? I'm using Julia, so it's ...
2
votes
1answer
43 views

Continuous joint entropy with fully dependent variable

Consider a variable $X$ with a continuous uniform distribution in the interval $[a,b]$ and a variable $Y$ that is fully dependent on $X$, i.e., $p(Y=y\ |\ X=x) = \delta (x-y)$, where $\delta$ is a ...
0
votes
2answers
40 views

Clean data and noisy data: which one has higher entropy?

I have a question regarding to the information theory. Between clean data and noisy data, which one has higher entropy? I think the noisy data has, am I right? But, noisy data does not have more ...
4
votes
0answers
80 views

Does Breiman's random forest use information gain or Gini index?

I would like to know if Breiman's random forest (random forest in R randomForest package) uses as a splitting criterion (criterion for attribute selection) information gain or Gini index? I tried to ...
3
votes
1answer
201 views

Calculating Diversity using normalized species counts

I am trying to calculate the change in diversity of species encountered throughout time. My dataset is comprised to detections of tagged fish for 48 consecutive weeks. My thought was to bin the ...
0
votes
1answer
92 views

Entropy of multivariate gaussian mixture random variable

Short: ${\bf X} \sim N({\bf 0},{\bf I}+{\bf I}_j)$; ${\bf I}_j\in S=\{I_j: I_j$ is diagonal and $ I_j \succeq 0\}, |S|=K$, and $j\sim U(1,K)$. What is $h({\bf X})$? What happens when ...
0
votes
2answers
57 views

Conditional entropy and Spearman's correlation based lag in time series

I have two time series A, B. Both are seasonal and B primarily is A driven( other temporal causes may exist). B-Red, A- Green I want to calculate lag of red series with respect to green as clearly, ...
0
votes
1answer
34 views

Transfer entropy on real, continuous data

Here is the goal and problem: I am trying to calculate a measure of coupling between real-valued, continuous oscillatory data. The data come from two people producing synchronized rhythmic ...
0
votes
0answers
14 views

Residuals as a percentage

Considering that still there is no suggestions, I assumed that my initial question is not fully understandable. I will try to ask same question in more abstract way, hoping it will come out as better ...
0
votes
1answer
83 views

Find entropy in WEKA

I am new in data mining so sorry for asking this kind of silly question. I am working on FAST feature selection algorithm and for that I need to find entropy of each attribute in dataset. But the ...
0
votes
0answers
32 views

R “Entropy” package gives weird KL divergence results

Using the R "entropy" package, I tried some KL divergence computations as a sanity check, but I'm getting weird results. For instance, shouldn't the following all be 2*log2(2)= 2 ? Instead, I'm ...
1
vote
1answer
84 views

How does the log(p(x,y)) normalize the pointwise mutual information?

I'm trying to understand the normalized form of pointwise mutual information. $npmi = \frac{pmi(x,y)}{log(p(x,y))}$ Why does the log joint probability normalize the pointwise mutual information to ...
0
votes
0answers
121 views

Entropy, Softmax and the derivative term in Backpropagation

I'm currently interested in using Cross Entropy Error when performing the BackPropagation algorithm for classification, where I use the Softmax Activation Function in my output layer. From what I ...
2
votes
0answers
34 views

Maximum Entropy with no index

This is a simpler problem than trying to solve, but have a feeling once get the methodology I can apply it to the harder problem. Let $ H(p)= -q \ln(q) - p \ln(p) $ be the entropy of the Bernoulli ...
0
votes
1answer
62 views

Sum of truncated normal distributions with other distributions (like uniform) that have partially common domains

I have some truncated normal distributions and some other distributions (like uniform distribution) that have partially common domains. I'd like to know how can I calculate the entropy for the sum of ...
1
vote
1answer
62 views

Tsallis and Renyi Normalized Entropy

I'm working with Shannon, Tsallis and Renyi entropies. I need to normalized these entropies for comparison purposes. In Shannon's entropy you need only to divide by the log of the number of bins. ...
1
vote
0answers
39 views

Propagation of uncertainty: entropy of multinomial

My goal is to estimate the entropy of a multinomial distribution, based on a single observation (a set of counts for each possible outcome). I also want to calculate the uncertainty in my estimated ...
1
vote
0answers
73 views

Decision tree with adaboost

Helllo! I'm currently learning the AdaBoost algorithm to use it with Decision Tree. I want to implement everything myself (that's the way I learn - implement everything from scratch and later use ...
4
votes
2answers
105 views

Is it possible to use SD instead of entropy?

While discussing about decision trees in class, my teacher touched upon the topic of entropy. I have understood the purpose of entropy (have not understood how the formula $H(X)= -\sum_{i}{p(x_i) \log ...
1
vote
1answer
44 views

Why to use Chi2 instead of accuracy in a decision tree?

Why many decision trees are using Chi2 or Information Gain Ratio to split the node when they can directly use accuracy, lift or AUC?
3
votes
0answers
63 views

Computation of the entropy of marginals?

I have implemented this paper: Efficient graph-based semi-supervised learning of structured tagging models In the last sentence of the section 4.2, the authors have mentioned another possible way of ...
5
votes
1answer
232 views

How to determine Forecastability of time series?

One of the important issues facing forecasters is if the given series can be forecasted or not ? I stumbled on an article entitled "Entropy as an A Priori Indicator of Forecastability" by Peter ...
3
votes
1answer
61 views

Entropy of Sum vs Difference of Random Variable

I am looking for a proof of the following fact Let $X$ and$ X'$ be i.i.d on $\{0,1,2\}$ (not necessarily uniform). Prove that $$H((X + X') \mod3) \leq H((X - X') \mod3)$$ where $H()$ is the ...
1
vote
1answer
315 views

Is there an R package to calculate differential entropy

I am trying to calculate differential entropy over my data. This is how a subset of my data set looks like : ...
0
votes
0answers
34 views

structural equation model based on generalized maximum entropy in R

I am trying to conduct an experiment based on generalized maximum entropy but I am not sure on how GME is different from maximum entropy. Can anybody tell me how to reparametized the SEM based on GME ...
2
votes
2answers
125 views

Why KL-Divergence uses “ln” in its formula?

I notice in KL-Divergence formula a $ln$ function is used: $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability ...
2
votes
1answer
91 views

Entropy stays the same with a larger distribution?

Based on the given definition of entropy, $H(P(X)) = -\sum_i P(x_i)log_2(P(x_i)$, it appears that if I have a distribution $P_1(x) = [\frac{1}{4},\frac{1}{4},\frac{1}{4},\frac{1}{4}]$ and another ...
0
votes
0answers
75 views

What is the relationship between Shannon Entropy and Approximate Entropy

I prefer a worded explanation because I am not a math expert. I know it Approximate Entropy is derived as an aproximation of Kolmogorov Entropry so the relationship between Kolmogorov Entropry and ...
1
vote
1answer
72 views

Why constrain mean and standard deviation when proving Gaussian is maximum differential entropy pdf?

I'm reading Bishop's Pattern Recognition and Machine Learning. In chapter 1.6: Information Theory (page 53) when trying to derive the maximum differential entropy pdf from the definition of continuous ...