A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

0
votes
1answer
21 views

Why is the log taken in the formula for weight of evidence?

Why is the logarithm used when calculating the weight of evidence (WOE)? For example, let bin i ($B_i$) have, 15% good 30% bad So good/bad = 0.5. Namely for each bad item there are 0.5 good in ...
1
vote
0answers
20 views

Relation between cross-entropy loss on softmax output and bits per base in DNA sequence compression

When training an autoencoder (softmax in outputlayer), for DNA sequence compression (by minimizing a cross-entropy loss function), how can you calculate the number of bits per base required for the ...
1
vote
0answers
12 views

Looking for derivation of Shannon entropy of binomial distribution

According to Wikipedia, the Shannon entropy of a binomial distribution is $\frac{1}{2}log_2 (2\pi*e* np(1-p)) + O(1/n)$ Does anyone know where I can find a derivation?
0
votes
0answers
28 views

Methodology to calculate Blackjack optimal strategy

Which methodology is best way to derive optimal blackjack strategy for player? I have come across some articles which suggest Markov chain. Is this the best way? Any reference/resource will be ...
1
vote
1answer
42 views

How is my definition of “information” different than Shannon's entropy?

I was trying to help some person here, but then I discovered that I actually almost know entropy (but not fully). Questions: Q1: how is my definition of information different than that of Claude ...
1
vote
1answer
29 views

Confidence bands for model averaged predictions of GLMMs

I use R with the MuMIn package for Multimodel inference. my global Model is ...
0
votes
0answers
19 views

Do not estimate model averaged parameters for mixed models?

Can anybody explain to me what is meant by "One word of warning, do not estimate model averaged parameters for mixed models! You can, however, model average they predictions of GLMM." at the end of ...
0
votes
1answer
40 views

Using Information theory with all possible models to select the best Model

I have a simple data set to find out about the effect of cultivation period length on soil organisms. The main factor of interest is age_class, a categorical variable defining the age of a field under ...
1
vote
0answers
47 views

Correct way of computing Shannon Entropy of a walk

Take for example a walk such as: ["school", "work", "home", "kindergarten", "home", "school", ...] # or simply [1, 2, 3, 4, 3, 1, ...] What's the correct way of ...
0
votes
0answers
33 views

orthogonal constraints in maxEnt methods

In a typical entropy maximization problem, in which we wish to maximize the entropy of a distribution subject to several moment constraints, how important is it that the moment constraints are ...
0
votes
0answers
25 views

Decision Trees: Why not this instead of Information Gain?

In Decision Trees one wants to say in which order one wants to put (splits on) Features in the tree. Say, for example we have two discretely valued Features F,G and the target Feature Y is binary ...
3
votes
0answers
19 views

Total Correlation with Renyi Entropy

The measure total correlation is defined making use of Shannon's entropy: $$ TC(X_1,\dots,X_m) = \sum_{i} H(X_i) - H(X_1,\dots,X_m) $$ This comes also with different names: e.g. multi information, or ...
0
votes
0answers
31 views

Can I increase Pointwise Mutual Information scores with a bandwidth or threshold?

I'm calculating PMI (Pointwise Mutual Information) wikipedia formula values between two time series x and y using the formula: $PMI(x, y) = \log( \frac{p(x,y)} {p(x) * p(y)} )$ where $p(x), p(y)$ ...
1
vote
0answers
19 views

Information entropy in a direct graph?

I have a directed graph, and each edge in the graph has a probability, representing certainty in the edge. How can I represent overall uncertainty in the network. I was thinking of using an ...
0
votes
0answers
17 views

Got an entropy-ish function for a multinomial distribution? Graph theory and Bayes net related

I have a discrete variable $X$ that can take on one of three states; $a$, $b$, and $c$. Thus it has two parameters $p_a = P(X = a)$ and $p_b = P(X = b)$, of course $P(X = c) = 1 - p_a - p_b$. I am ...
1
vote
2answers
59 views

Is $I(X,Y;Z)$ connected with $I(X;Z)$ and $I(Y;Z)$?

Let $I(X,Y;Z)$ be the mutual information between the tuple $X,Y$ and the variable $Z$: $$I(X,Y;Z) = H(X,Y) - H(X,Y|Z)$$ From the data processing inequality it follows trivially that: $$I(X;Z) \le ...
1
vote
0answers
19 views

What is the best way to decide bin size for computing Entropy or Mutual Information?

I have a continuous distribution that I was thinking of binning for computing MI and H. I often arbitrarily decide on bin size. Is there a general consensus on how to set bin size and number? Thanks ...
2
votes
0answers
39 views

Limit of quantized entropy

Consider the continuous random variable $X \in \mathbb R$ and the deterministic function $Y = f(X)$, $f: \mathbb R \to \mathbb R$. For the conditional entropy we have $$H(Y|X) = 0.$$ Does this imply ...
0
votes
0answers
9 views

Evaluate information lost of two different estimators using AIC?

I thinking about following issue. I have two type of estimators. 1) The first one is classic glm so the final rate is predicted values from glm. The second estimator use following method. At ...
0
votes
0answers
18 views

How to compare 2 estimations of a probability distribution?

I am using a program that returns a probability estimation Q to predict a value. I have access to the real probability distribution P that I try to estimate, from the dataset. I can compute the ...
0
votes
0answers
7 views

How to design a testing rule to fastest differentiate skilled from unskilled experts?

The problem is as follows. There are $n$ securities. In each period $t$, a random return vector $R_t = \alpha_t + \epsilon_t$ will be drawn, where $\alpha_t \sim N(0,\Sigma_\alpha)$ and $\epsilon_t ...
1
vote
1answer
57 views

How to derive the probability function from information entropy?

How to derive the probability function from information entropy? $H(X)=\sum P(x_i)I(x_i)$ $H(X)=\int P(x)I(x) dx$ Can we get the equation in the form of P=f(H)?
0
votes
0answers
13 views

Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
0
votes
1answer
26 views

Computing the Interaction gain. Is there an Error in the infotheo package in R?

In order to implementing a certain feature selection method for a classification problem I need to estimate the the interaction the interaction gain between two features and the target variable which ...
9
votes
2answers
260 views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
1
vote
1answer
45 views

Is there an intuitive interpretation of mutual information values (bits, nits)?

I understand how mutual information is calculated, and what it is addressing: how much the distribution of one variable changes conditional on the value of another variable. But I don't really ...
7
votes
0answers
73 views

References that justify use of Gaussian Mixtures

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much ...
0
votes
0answers
12 views

optimization problem rewrite help

From my understanding, the transformation between probability and information content for d=2 is as follows: Information = -log_2 (Probability) If I have an optimization problem that is written in ...
2
votes
1answer
66 views

What makes something a “Probability”?

What makes it legitimate to say a set of values are probabilities? (I need an answer for a formalisation reasons.) Let me give a simple example. Let's assume that I have a number of letters and each ...
10
votes
2answers
272 views

Different AIC definitions

From Wikipedia there is a definition of Akaike's Information Criterion (AIC) as $ AIC = 2k -2 \log L $, where $k$ is the number of parameters and $\log L$ is the log-likelihood of the model. However, ...
4
votes
0answers
62 views

Find the input/output probability distributions that realize the capacity of communications channel?

Consider two random variables $X,Y$. For simplicity, they are discrete and finite. Let $Q(y|x)$ be the conditional probability of $Y$ given $X$. Their mutual information is defined as: ...
4
votes
1answer
34 views

Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
0
votes
0answers
28 views

“Pairwise dependence probability”

From pg 9 of [1]: The notion of dependence between two variables A and B is taken to be mutual information; the amount of evidence for dependence is then the probability that the mutual ...
0
votes
0answers
19 views

When can conditional mutual information be decomposed as a sum?

Let $X$, $Y$, $Z$ be discrete random variables, each on his own support. What are the necessary conditions to be able to write the following: $$I(X;Y|Z) = \sum_z p(z) \cdot I(X;Y|Z=z)$$ Isn't this ...
4
votes
0answers
102 views

Expected and observed Fisher information?

Studying asymptotics, I bumped into the concept of Observed Fisher Information, as a way to compute Fisher Information when the parameter $\theta$ is unknown. I am also aware that it is related in ...
2
votes
0answers
28 views

Entropy measure for multiple node data splitting with post-normalization constraint

Let me introduce my problem with a simple example. Let's say that we have two different classes $C_0$ and $C_1$ and we have one node $S$ that has the following elements of each class: $S = ...
4
votes
1answer
75 views

Weighted entropy as a measure of diversity

Suppose that you are a company manager and you are looking for a statistical measure that defines the international reputation of your company. So, you collect data on your clients and the countries ...
0
votes
0answers
17 views

How do we characterize model richness under regularization?

Say we have N parameters in a linear regression model. Often, we add priors to say we expect those parameters to be 0, and then fit to some data points. (I've seen this used with "infinite" parameter ...
3
votes
0answers
43 views

Is it possible to get a negative infinite differential entropy without delta function and limit?

If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?
1
vote
0answers
331 views

CI's from full model-averaged coefficients from model.avg (MuMIn package) [closed]

I am currently trying to get model-averaged estimates (and confidence intervals) for a GLMM I am running. After obtaining a full set of candidate models using the ...
0
votes
0answers
40 views

What is the information-theoretical lower bound for estimating $E[f(X)]$

Given i.i.d. realizations $X_1,\ldots,X_n$ of a random variable $X$, what is the information-theoretical lower bound of the convergence rate for estimating $E[f(X)]$? Here $f$ is a function from ...
2
votes
2answers
220 views

Number of bins when computing mutual information

I want to quantify the relationship between two variables, A and B, using mutual information. The way to compute it is by binning the observations (see example Python code below). However, what ...
0
votes
2answers
86 views

Error in mutual information when using a subset

I want to compute the mutual information for ~4000 different pairs, where each pair contains two vectors. Each of these vectors hold 100000 observations, making this computation very computationally ...
0
votes
1answer
35 views

Mutual Information based Correlation Measures that are Robust to Compositional Data

Are there mutual-information based correlation measures that are robust to compositional data? It is my belief that many of these methods (e.g., distance correlation, transfer entropy, MIC, ...) may ...
1
vote
0answers
30 views

Entropy rate for continuous variables?

As far as I know, the entropy rate of a random process is defined as $$ h = \lim_{n \rightarrow \infty} \frac{1}{n}H(X_1, ..., H_n) $$ In conventional, finite-alphabet information theory the entropy ...
0
votes
1answer
69 views

Is weight of evidence and information value a technique of dimension reduction

I am trying to understand the concept of weight of evidence and information value. From what I understand, it is a variable reduction technique where we only use variables with IV > 0.5 in the model. ...
0
votes
0answers
17 views

Comparing two features capturing the same aspect but calculated differently?

In a classification problem I am trying to solve, one of the input (independent) features can be calculated by using two alternative calculation methods. One way of comparing the two methods is to see ...
2
votes
0answers
58 views

joint differential entropy $h(X,Y)$, when $Y=g(X)$

It's well known that if X & Y are discrete random variables X & Y (r.v.s), and $Y=g(X)$, then $$H(X,Y)=H(X)+H(Y\mid X)=H(X),$$ where the last equality is due to $H(Y\mid X)=0.$ It also has ...
2
votes
1answer
42 views

Mutual information equals conditional mutual information

Consider three random variables $X,Y,Z$. It is standard that $I(X,Y|Z)=0$ if and only if $X,Y$ are conditionally independent given $Z$. If instead we require $I(X,Y|Z)=I(X,Y)$, what do we get? What ...
2
votes
1answer
147 views

Relationship between least-squares regression and information theory

Is there a well-known relationship between least-squares regression and information theory? I've just started reading about information theory. It seems almost trivial to say that the regression ...