A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

3
votes
1answer
15 views

Is it valid to reduce the AICc penalty for multiple variables, when some variables should be grouped?

I have a data set of mean trait values for each of 18 populations, and want to test whether several ecological variables are related to variation in traits. I'm using the corrected Akaike information ...
0
votes
0answers
12 views

How to deal with missing data when calculating Information Gain

While working on a neural network for classification problem I'm dealing with huge number of possible features and information gain seems like a good way to narrow them down (there are hundreds of ...
0
votes
1answer
27 views

In feature selection, are there any rules on choosing metrics to mesure relevance? (MI / Fisher score / correlation coefficient, etc)

This is a rather general question. If the question is vague and hard to answer in a few lines, I'd be happy if someone just point me to some readings. Thanks in Advance. I am working on a multi-class ...
1
vote
1answer
36 views

How to calculate the information loss of PCA?

How would we calculate the information loss of reducing dimensions using PCA ? Would it be the amount of variance loss if we skip certain eigenvectors after the PCA ?
8
votes
0answers
55 views

What does the Akaike Information Criterion (AIC) score of a model mean?

I have seen some questions here about what it means in layman terms, but these are too layman for for my purpose here. I am trying to mathematically understand what does the AIC score mean. But at ...
6
votes
1answer
364 views

Interpretation of the entropy with a coding length?

There is this interpretation of the entropy $-\sum_i p_i \log_2 p_i$ as the average length (in bits) per character when using an optimal encoding of a message. Now, if we use the simple 3-letter case ...
1
vote
2answers
42 views

Transfer entropy value between 0 and 1

Given two variables, X and Y, there is a way of obtaining a Mutual Information value between 0 and 1 by: MI_normalised=MI_original/sqrt(H(X)*H(Y)); where H(X) ...
0
votes
0answers
9 views

Is Lehmann-informativeness transitive?

Experiments $\mathbf{E}$ and $\mathbf{F}$ are defines as follows: An experiment $\mathbf{E}$ is a random quantity $X$ and a family $\mathbf{P} = \{P_\theta,\,\theta\in\Omega\}$ of possible ...
0
votes
1answer
31 views

Mutual information formula

What is the formula for finding mutual information (MI) feature to feature? Is this formula correct $$ \mathrm{MI}=p(t_1,t_2)\log_2\left(\frac{p(t_1,t_2)}{p(t_1)p(t_2)}\right) + (1-p(t_1,t_2)) \log_2\...
1
vote
0answers
12 views

How to decide what is the relevant group in a precision and recall computation?

One of the most famous measurements for an information retrieval system is to compute its precision and recall. For both cases, we need to compute the number of total relevant documents and compare it ...
0
votes
0answers
22 views

Question about the Probability of Error for Joint-Typicality Tests

Given a set of codewords $\boldsymbol{x}_i$ with $i=1,\cdots,2^{nR}$ where $R$ is the rate of the code. The codewords are transmitted over a Gaussian channel $Y = X + W$ with $X\sim\mathcal{N}(0,A^2)$ ...
0
votes
1answer
27 views

Why is the log taken in the formula for weight of evidence?

Why is the logarithm used when calculating the weight of evidence (WOE)? For example, let bin i ($B_i$) have, 15% good 30% bad So good/bad = 0.5. Namely for each bad item there are 0.5 good in $...
1
vote
0answers
31 views

Relation between cross-entropy loss on softmax output and bits per base in DNA sequence compression

When training an autoencoder (softmax in outputlayer), for DNA sequence compression (by minimizing a cross-entropy loss function), how can you calculate the number of bits per base required for the ...
1
vote
0answers
13 views

Looking for derivation of Shannon entropy of binomial distribution

According to Wikipedia, the Shannon entropy of a binomial distribution is $\frac{1}{2}log_2 (2\pi*e* np(1-p)) + O(1/n)$ Does anyone know where I can find a derivation?
0
votes
0answers
29 views

Methodology to calculate Blackjack optimal strategy

Which methodology is best way to derive optimal blackjack strategy for player? I have come across some articles which suggest Markov chain. Is this the best way? Any reference/resource will be ...
1
vote
1answer
43 views

How is my definition of “information” different than Shannon's entropy?

I was trying to help some person here, but then I discovered that I actually almost know entropy (but not fully). Questions: Q1: how is my definition of information different than that of Claude ...
1
vote
1answer
43 views

Confidence bands for model averaged predictions of GLMMs

I use R with the MuMIn package for Multimodel inference. my global Model is ...
0
votes
0answers
29 views

Do not estimate model averaged parameters for mixed models?

Can anybody explain to me what is meant by "One word of warning, do not estimate model averaged parameters for mixed models! You can, however, model average they predictions of GLMM." at the end of ...
0
votes
1answer
43 views

Using Information theory with all possible models to select the best Model

I have a simple data set to find out about the effect of cultivation period length on soil organisms. The main factor of interest is age_class, a categorical variable defining the age of a field under ...
1
vote
1answer
110 views

Correct way of computing Shannon Entropy of a walk

Take for example a walk such as: ["school", "work", "home", "kindergarten", "home", "school", ...] # or simply [1, 2, 3, 4, 3, 1, ...] What's the correct way of ...
0
votes
0answers
34 views

orthogonal constraints in maxEnt methods

In a typical entropy maximization problem, in which we wish to maximize the entropy of a distribution subject to several moment constraints, how important is it that the moment constraints are ...
0
votes
0answers
27 views

Decision Trees: Why not this instead of Information Gain?

In Decision Trees one wants to say in which order one wants to put (splits on) Features in the tree. Say, for example we have two discretely valued Features F,G and the target Feature Y is binary (i.e....
3
votes
0answers
21 views

Total Correlation with Renyi Entropy

The measure total correlation is defined making use of Shannon's entropy: $$ TC(X_1,\dots,X_m) = \sum_{i} H(X_i) - H(X_1,\dots,X_m) $$ This comes also with different names: e.g. multi information, or ...
0
votes
0answers
31 views

Can I increase Pointwise Mutual Information scores with a bandwidth or threshold?

I'm calculating PMI (Pointwise Mutual Information) wikipedia formula values between two time series x and y using the formula: $PMI(x, y) = \log( \frac{p(x,y)} {p(x) * p(y)} )$ where $p(x), p(y)$ ...
1
vote
0answers
22 views

Information entropy in a direct graph?

I have a directed graph, and each edge in the graph has a probability, representing certainty in the edge. How can I represent overall uncertainty in the network. I was thinking of using an ...
0
votes
0answers
30 views

Got an entropy-ish function for a multinomial distribution? Graph theory and Bayes net related

I have a discrete variable $X$ that can take on one of three states; $a$, $b$, and $c$. Thus it has two parameters $p_a = P(X = a)$ and $p_b = P(X = b)$, of course $P(X = c) = 1 - p_a - p_b$. I am ...
1
vote
2answers
59 views

Is $I(X,Y;Z)$ connected with $I(X;Z)$ and $I(Y;Z)$?

Let $I(X,Y;Z)$ be the mutual information between the tuple $X,Y$ and the variable $Z$: $$I(X,Y;Z) = H(X,Y) - H(X,Y|Z)$$ From the data processing inequality it follows trivially that: $$I(X;Z) \le I(...
1
vote
0answers
20 views

What is the best way to decide bin size for computing Entropy or Mutual Information?

I have a continuous distribution that I was thinking of binning for computing MI and H. I often arbitrarily decide on bin size. Is there a general consensus on how to set bin size and number? Thanks ...
2
votes
0answers
39 views

Limit of quantized entropy

Consider the continuous random variable $X \in \mathbb R$ and the deterministic function $Y = f(X)$, $f: \mathbb R \to \mathbb R$. For the conditional entropy we have $$H(Y|X) = 0.$$ Does this imply $$...
0
votes
0answers
10 views

Evaluate information lost of two different estimators using AIC?

I thinking about following issue. I have two type of estimators. 1) The first one is classic glm so the final rate is predicted values from glm. The second estimator use following method. At ...
0
votes
0answers
18 views

How to compare 2 estimations of a probability distribution?

I am using a program that returns a probability estimation Q to predict a value. I have access to the real probability distribution P that I try to estimate, from the dataset. I can compute the ...
0
votes
0answers
7 views

How to design a testing rule to fastest differentiate skilled from unskilled experts?

The problem is as follows. There are $n$ securities. In each period $t$, a random return vector $R_t = \alpha_t + \epsilon_t$ will be drawn, where $\alpha_t \sim N(0,\Sigma_\alpha)$ and $\epsilon_t \...
1
vote
1answer
59 views

How to derive the probability function from information entropy?

How to derive the probability function from information entropy? $H(X)=\sum P(x_i)I(x_i)$ $H(X)=\int P(x)I(x) dx$ Can we get the equation in the form of P=f(H)?
0
votes
0answers
15 views

Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
0
votes
1answer
29 views

Computing the Interaction gain. Is there an Error in the infotheo package in R?

In order to implementing a certain feature selection method for a classification problem I need to estimate the the interaction the interaction gain between two features and the target variable which ...
11
votes
2answers
271 views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
1
vote
1answer
68 views

Is there an intuitive interpretation of mutual information values (bits, nits)?

I understand how mutual information is calculated, and what it is addressing: how much the distribution of one variable changes conditional on the value of another variable. But I don't really ...
9
votes
2answers
107 views

References that justify use of Gaussian Mixtures

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much ...
0
votes
0answers
13 views

optimization problem rewrite help

From my understanding, the transformation between probability and information content for d=2 is as follows: Information = -log_2 (Probability) If I have an optimization problem that is written in ...
2
votes
1answer
67 views

What makes something a “Probability”?

What makes it legitimate to say a set of values are probabilities? (I need an answer for a formalisation reasons.) Let me give a simple example. Let's assume that I have a number of letters and each ...
10
votes
2answers
279 views

Different AIC definitions

From Wikipedia there is a definition of Akaike's Information Criterion (AIC) as $ AIC = 2k -2 \log L $, where $k$ is the number of parameters and $\log L$ is the log-likelihood of the model. However, ...
4
votes
0answers
62 views

Find the input/output probability distributions that realize the capacity of communications channel?

Consider two random variables $X,Y$. For simplicity, they are discrete and finite. Let $Q(y|x)$ be the conditional probability of $Y$ given $X$. Their mutual information is defined as: $$I=H(Y)-H(Y|X)...
4
votes
1answer
35 views

Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
0
votes
0answers
29 views

“Pairwise dependence probability”

From pg 9 of [1]: The notion of dependence between two variables A and B is taken to be mutual information; the amount of evidence for dependence is then the probability that the mutual ...
0
votes
0answers
20 views

When can conditional mutual information be decomposed as a sum?

Let $X$, $Y$, $Z$ be discrete random variables, each on his own support. What are the necessary conditions to be able to write the following: $$I(X;Y|Z) = \sum_z p(z) \cdot I(X;Y|Z=z)$$ Isn't this ...
4
votes
0answers
112 views

Expected and observed Fisher information?

Studying asymptotics, I bumped into the concept of Observed Fisher Information, as a way to compute Fisher Information when the parameter $\theta$ is unknown. I am also aware that it is related in ...
2
votes
0answers
30 views

Entropy measure for multiple node data splitting with post-normalization constraint

Let me introduce my problem with a simple example. Let's say that we have two different classes $C_0$ and $C_1$ and we have one node $S$ that has the following elements of each class: $S = \{1000,...
4
votes
1answer
78 views

Weighted entropy as a measure of diversity

Suppose that you are a company manager and you are looking for a statistical measure that defines the international reputation of your company. So, you collect data on your clients and the countries ...
3
votes
0answers
45 views

Is it possible to get a negative infinite differential entropy without delta function and limit?

If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?
1
vote
0answers
431 views

CI's from full model-averaged coefficients from model.avg (MuMIn package) [closed]

I am currently trying to get model-averaged estimates (and confidence intervals) for a GLMM I am running. After obtaining a full set of candidate models using the ...