A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

-1
votes
0answers
10 views

Implementation of Meila's VI criterion in Python? [closed]

I am doing some clustering experiments and came across this paper by Marina Meilă in the Journal of Multivar Statistics, where she presents a very interesting metric for evaluating clusterings called ...
0
votes
0answers
9 views

Is Lehmann-informativeness transitive?

Experiments $\mathbf{E}$ and $\mathbf{F}$ are defines as follows: An experiment $\mathbf{E}$ is a random quantity $X$ and a family $\mathbf{P} = \{P_\theta,\,\theta\in\Omega\}$ of possible ...
0
votes
1answer
29 views

Mutual information formula

What is the formula for finding mutual information (MI) feature to feature? Is this formula correct $$ \mathrm{MI}=p(t_1,t_2)\log_2\left(\frac{p(t_1,t_2)}{p(t_1)p(t_2)}\right) + (1-p(t_1,t_2)) ...
1
vote
0answers
12 views

How to decide what is the relevant group in a precision and recall computation?

One of the most famous measurements for an information retrieval system is to compute its precision and recall. For both cases, we need to compute the number of total relevant documents and compare it ...
0
votes
0answers
22 views

Question about the Probability of Error for Joint-Typicality Tests

Given a set of codewords $\boldsymbol{x}_i$ with $i=1,\cdots,2^{nR}$ where $R$ is the rate of the code. The codewords are transmitted over a Gaussian channel $Y = X + W$ with $X\sim\mathcal{N}(0,A^2)$ ...
0
votes
1answer
23 views

Why is the log taken in the formula for weight of evidence?

Why is the logarithm used when calculating the weight of evidence (WOE)? For example, let bin i ($B_i$) have, 15% good 30% bad So good/bad = 0.5. Namely for each bad item there are 0.5 good in ...
1
vote
0answers
27 views

Relation between cross-entropy loss on softmax output and bits per base in DNA sequence compression

When training an autoencoder (softmax in outputlayer), for DNA sequence compression (by minimizing a cross-entropy loss function), how can you calculate the number of bits per base required for the ...
1
vote
0answers
13 views

Looking for derivation of Shannon entropy of binomial distribution

According to Wikipedia, the Shannon entropy of a binomial distribution is $\frac{1}{2}log_2 (2\pi*e* np(1-p)) + O(1/n)$ Does anyone know where I can find a derivation?
0
votes
0answers
28 views

Methodology to calculate Blackjack optimal strategy

Which methodology is best way to derive optimal blackjack strategy for player? I have come across some articles which suggest Markov chain. Is this the best way? Any reference/resource will be ...
1
vote
1answer
43 views

How is my definition of “information” different than Shannon's entropy?

I was trying to help some person here, but then I discovered that I actually almost know entropy (but not fully). Questions: Q1: how is my definition of information different than that of Claude ...
1
vote
1answer
37 views

Confidence bands for model averaged predictions of GLMMs

I use R with the MuMIn package for Multimodel inference. my global Model is ...
0
votes
0answers
23 views

Do not estimate model averaged parameters for mixed models?

Can anybody explain to me what is meant by "One word of warning, do not estimate model averaged parameters for mixed models! You can, however, model average they predictions of GLMM." at the end of ...
0
votes
1answer
42 views

Using Information theory with all possible models to select the best Model

I have a simple data set to find out about the effect of cultivation period length on soil organisms. The main factor of interest is age_class, a categorical variable defining the age of a field under ...
1
vote
0answers
66 views

Correct way of computing Shannon Entropy of a walk

Take for example a walk such as: ["school", "work", "home", "kindergarten", "home", "school", ...] # or simply [1, 2, 3, 4, 3, 1, ...] What's the correct way of ...
0
votes
0answers
33 views

orthogonal constraints in maxEnt methods

In a typical entropy maximization problem, in which we wish to maximize the entropy of a distribution subject to several moment constraints, how important is it that the moment constraints are ...
0
votes
0answers
26 views

Decision Trees: Why not this instead of Information Gain?

In Decision Trees one wants to say in which order one wants to put (splits on) Features in the tree. Say, for example we have two discretely valued Features F,G and the target Feature Y is binary ...
3
votes
0answers
20 views

Total Correlation with Renyi Entropy

The measure total correlation is defined making use of Shannon's entropy: $$ TC(X_1,\dots,X_m) = \sum_{i} H(X_i) - H(X_1,\dots,X_m) $$ This comes also with different names: e.g. multi information, or ...
0
votes
0answers
31 views

Can I increase Pointwise Mutual Information scores with a bandwidth or threshold?

I'm calculating PMI (Pointwise Mutual Information) wikipedia formula values between two time series x and y using the formula: $PMI(x, y) = \log( \frac{p(x,y)} {p(x) * p(y)} )$ where $p(x), p(y)$ ...
1
vote
0answers
19 views

Information entropy in a direct graph?

I have a directed graph, and each edge in the graph has a probability, representing certainty in the edge. How can I represent overall uncertainty in the network. I was thinking of using an ...
0
votes
0answers
19 views

Got an entropy-ish function for a multinomial distribution? Graph theory and Bayes net related

I have a discrete variable $X$ that can take on one of three states; $a$, $b$, and $c$. Thus it has two parameters $p_a = P(X = a)$ and $p_b = P(X = b)$, of course $P(X = c) = 1 - p_a - p_b$. I am ...
1
vote
2answers
59 views

Is $I(X,Y;Z)$ connected with $I(X;Z)$ and $I(Y;Z)$?

Let $I(X,Y;Z)$ be the mutual information between the tuple $X,Y$ and the variable $Z$: $$I(X,Y;Z) = H(X,Y) - H(X,Y|Z)$$ From the data processing inequality it follows trivially that: $$I(X;Z) \le ...
1
vote
0answers
19 views

What is the best way to decide bin size for computing Entropy or Mutual Information?

I have a continuous distribution that I was thinking of binning for computing MI and H. I often arbitrarily decide on bin size. Is there a general consensus on how to set bin size and number? Thanks ...
2
votes
0answers
39 views

Limit of quantized entropy

Consider the continuous random variable $X \in \mathbb R$ and the deterministic function $Y = f(X)$, $f: \mathbb R \to \mathbb R$. For the conditional entropy we have $$H(Y|X) = 0.$$ Does this imply ...
0
votes
0answers
10 views

Evaluate information lost of two different estimators using AIC?

I thinking about following issue. I have two type of estimators. 1) The first one is classic glm so the final rate is predicted values from glm. The second estimator use following method. At ...
0
votes
0answers
18 views

How to compare 2 estimations of a probability distribution?

I am using a program that returns a probability estimation Q to predict a value. I have access to the real probability distribution P that I try to estimate, from the dataset. I can compute the ...
0
votes
0answers
7 views

How to design a testing rule to fastest differentiate skilled from unskilled experts?

The problem is as follows. There are $n$ securities. In each period $t$, a random return vector $R_t = \alpha_t + \epsilon_t$ will be drawn, where $\alpha_t \sim N(0,\Sigma_\alpha)$ and $\epsilon_t ...
1
vote
1answer
58 views

How to derive the probability function from information entropy?

How to derive the probability function from information entropy? $H(X)=\sum P(x_i)I(x_i)$ $H(X)=\int P(x)I(x) dx$ Can we get the equation in the form of P=f(H)?
0
votes
0answers
15 views

Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
0
votes
1answer
26 views

Computing the Interaction gain. Is there an Error in the infotheo package in R?

In order to implementing a certain feature selection method for a classification problem I need to estimate the the interaction the interaction gain between two features and the target variable which ...
10
votes
2answers
263 views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
1
vote
1answer
51 views

Is there an intuitive interpretation of mutual information values (bits, nits)?

I understand how mutual information is calculated, and what it is addressing: how much the distribution of one variable changes conditional on the value of another variable. But I don't really ...
9
votes
2answers
97 views

References that justify use of Gaussian Mixtures

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much ...
0
votes
0answers
13 views

optimization problem rewrite help

From my understanding, the transformation between probability and information content for d=2 is as follows: Information = -log_2 (Probability) If I have an optimization problem that is written in ...
2
votes
1answer
66 views

What makes something a “Probability”?

What makes it legitimate to say a set of values are probabilities? (I need an answer for a formalisation reasons.) Let me give a simple example. Let's assume that I have a number of letters and each ...
10
votes
2answers
274 views

Different AIC definitions

From Wikipedia there is a definition of Akaike's Information Criterion (AIC) as $ AIC = 2k -2 \log L $, where $k$ is the number of parameters and $\log L$ is the log-likelihood of the model. However, ...
4
votes
0answers
62 views

Find the input/output probability distributions that realize the capacity of communications channel?

Consider two random variables $X,Y$. For simplicity, they are discrete and finite. Let $Q(y|x)$ be the conditional probability of $Y$ given $X$. Their mutual information is defined as: ...
4
votes
1answer
35 views

Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
0
votes
0answers
29 views

“Pairwise dependence probability”

From pg 9 of [1]: The notion of dependence between two variables A and B is taken to be mutual information; the amount of evidence for dependence is then the probability that the mutual ...
0
votes
0answers
20 views

When can conditional mutual information be decomposed as a sum?

Let $X$, $Y$, $Z$ be discrete random variables, each on his own support. What are the necessary conditions to be able to write the following: $$I(X;Y|Z) = \sum_z p(z) \cdot I(X;Y|Z=z)$$ Isn't this ...
4
votes
0answers
110 views

Expected and observed Fisher information?

Studying asymptotics, I bumped into the concept of Observed Fisher Information, as a way to compute Fisher Information when the parameter $\theta$ is unknown. I am also aware that it is related in ...
2
votes
0answers
28 views

Entropy measure for multiple node data splitting with post-normalization constraint

Let me introduce my problem with a simple example. Let's say that we have two different classes $C_0$ and $C_1$ and we have one node $S$ that has the following elements of each class: $S = ...
4
votes
1answer
78 views

Weighted entropy as a measure of diversity

Suppose that you are a company manager and you are looking for a statistical measure that defines the international reputation of your company. So, you collect data on your clients and the countries ...
3
votes
0answers
44 views

Is it possible to get a negative infinite differential entropy without delta function and limit?

If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?
1
vote
0answers
385 views

CI's from full model-averaged coefficients from model.avg (MuMIn package) [closed]

I am currently trying to get model-averaged estimates (and confidence intervals) for a GLMM I am running. After obtaining a full set of candidate models using the ...
0
votes
0answers
40 views

What is the information-theoretical lower bound for estimating $E[f(X)]$

Given i.i.d. realizations $X_1,\ldots,X_n$ of a random variable $X$, what is the information-theoretical lower bound of the convergence rate for estimating $E[f(X)]$? Here $f$ is a function from ...
2
votes
2answers
263 views

Number of bins when computing mutual information

I want to quantify the relationship between two variables, A and B, using mutual information. The way to compute it is by binning the observations (see example Python code below). However, what ...
0
votes
2answers
88 views

Error in mutual information when using a subset

I want to compute the mutual information for ~4000 different pairs, where each pair contains two vectors. Each of these vectors hold 100000 observations, making this computation very computationally ...
0
votes
1answer
36 views

Mutual Information based Correlation Measures that are Robust to Compositional Data

Are there mutual-information based correlation measures that are robust to compositional data? It is my belief that many of these methods (e.g., distance correlation, transfer entropy, MIC, ...) may ...
1
vote
0answers
31 views

Entropy rate for continuous variables?

As far as I know, the entropy rate of a random process is defined as $$ h = \lim_{n \rightarrow \infty} \frac{1}{n}H(X_1, ..., H_n) $$ In conventional, finite-alphabet information theory the entropy ...
0
votes
1answer
73 views

Is weight of evidence and information value a technique of dimension reduction

I am trying to understand the concept of weight of evidence and information value. From what I understand, it is a variable reduction technique where we only use variables with IV > 0.5 in the model. ...