A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in ...

learn more… | top users | synonyms

0
votes
0answers
5 views

Correct Terminology of Information Gain

I'm trying to correctly understand terminology of information gain, entropy and Gini impurity. Do I understand it correctly in this way? Entropy change and Gini impurity are both "just" metric of ...
0
votes
1answer
22 views

Computing the Interaction gain. Is there an Error in the infotheo package in R?

In order to implementing a certain feature selection method for a classification problem I need to estimate the the interaction the interaction gain between two features and the target variable which ...
9
votes
2answers
214 views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
1
vote
1answer
23 views

Is there an intuitive interpretation of mutual information values (bits, nits)?

I understand how mutual information is calculated, and what it is addressing: how much the distribution of one variable changes conditional on the value of another variable. But I don't really ...
6
votes
0answers
45 views

References that justify use of Gaussian Mixtures

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much ...
0
votes
0answers
11 views

optimization problem rewrite help

From my understanding, the transformation between probability and information content for d=2 is as follows: Information = -log_2 (Probability) If I have an optimization problem that is written in ...
2
votes
1answer
64 views

What makes something a “Probability”?

What makes it legitimate to say a set of values are probabilities? (I need an answer for a formalisation reasons.) Let me give a simple example. Let's assume that I have a number of letters and each ...
10
votes
2answers
257 views

Different AIC definitions

From Wikipedia there is a definition of Akaike's Information Criterion (AIC) as $ AIC = 2k -2 \log L $, where $k$ is the number of parameters and $\log L$ is the log-likelihood of the model. However, ...
4
votes
0answers
60 views

Find the input/output probability distributions that realize the capacity of communications channel?

Consider two random variables $X,Y$. For simplicity, they are discrete and finite. Let $Q(y|x)$ be the conditional probability of $Y$ given $X$. Their mutual information is defined as: ...
4
votes
1answer
29 views

Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
0
votes
0answers
28 views

“Pairwise dependence probability”

From pg 9 of [1]: The notion of dependence between two variables A and B is taken to be mutual information; the amount of evidence for dependence is then the probability that the mutual ...
0
votes
0answers
14 views

When can conditional mutual information be decomposed as a sum?

Let $X$, $Y$, $Z$ be discrete random variables, each on his own support. What are the necessary conditions to be able to write the following: $$I(X;Y|Z) = \sum_z p(z) \cdot I(X;Y|Z=z)$$ Isn't this ...
3
votes
0answers
65 views

Expected and observed Fisher information?

Studying asymptotics, I bumped into the concept of Observed Fisher Information, as a way to compute Fisher Information when the parameter $\theta$ is unknown. I am also aware that it is related in ...
2
votes
0answers
17 views

Entropy measure for multiple node data splitting with post-normalization constraint

Let me introduce my problem with a simple example. Let's say that we have two different classes $C_0$ and $C_1$ and we have one node $S$ that has the following elements of each class: $S = ...
4
votes
1answer
52 views

Weighted entropy as a measure of diversity

Suppose that you are a company manager and you are looking for a statistical measure that defines the international reputation of your company. So, you collect data on your clients and the countries ...
0
votes
0answers
12 views

How do we characterize model richness under regularization?

Say we have N parameters in a linear regression model. Often, we add priors to say we expect those parameters to be 0, and then fit to some data points. (I've seen this used with "infinite" parameter ...
2
votes
0answers
37 views

Is it possible to get a negative infinite differential entropy without delta function and limit?

If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?
1
vote
0answers
156 views

CI's from full model-averaged coefficients from model.avg (MuMIn package) [closed]

I am currently trying to get model-averaged estimates (and confidence intervals) for a GLMM I am running. After obtaining a full set of candidate models using the ...
0
votes
0answers
33 views

What is the information-theoretical lower bound for estimating $E[f(X)]$

Given i.i.d. realizations $X_1,\ldots,X_n$ of $X$, what is the information-theoretical lower bound of the convergence rate for estimating $E[f(X)]$? Here $f$ is a function from $[0,1]$ to $[0,1]$. ...
2
votes
2answers
108 views

Number of bins when computing mutual information

I want to quantify the relationship between two variables, A and B, using mutual information. The way to compute it is by binning the observations (see example Python code below). However, what ...
0
votes
2answers
79 views

Error in mutual information when using a subset

I want to compute the mutual information for ~4000 different pairs, where each pair contains two vectors. Each of these vectors hold 100000 observations, making this computation very computationally ...
0
votes
1answer
28 views

Mutual Information based Correlation Measures that are Robust to Compositional Data

Are there mutual-information based correlation measures that are robust to compositional data? It is my belief that many of these methods (e.g., distance correlation, transfer entropy, MIC, ...) may ...
1
vote
0answers
24 views

Entropy rate for continuous variables?

As far as I know, the entropy rate of a random process is defined as $$ h = \lim_{n \rightarrow \infty} \frac{1}{n}H(X_1, ..., H_n) $$ In conventional, finite-alphabet information theory the entropy ...
0
votes
1answer
43 views

Is weight of evidence and information value a technique of dimension reduction

I am trying to understand the concept of weight of evidence and information value. From what I understand, it is a variable reduction technique where we only use variables with IV > 0.5 in the model. ...
0
votes
0answers
17 views

Comparing two features capturing the same aspect but calculated differently?

In a classification problem I am trying to solve, one of the input (independent) features can be calculated by using two alternative calculation methods. One way of comparing the two methods is to see ...
2
votes
0answers
49 views

joint differential entropy $h(X,Y)$, when $Y=g(X)$

It's well known that if X & Y are discrete random variables X & Y (r.v.s), and $Y=g(X)$, then $$H(X,Y)=H(X)+H(Y\mid X)=H(X),$$ where the last equality is due to $H(Y\mid X)=0.$ It also has ...
2
votes
1answer
35 views

Mutual information equals conditional mutual information

Consider three random variables $X,Y,Z$. It is standard that $I(X,Y|Z)=0$ if and only if $X,Y$ are conditionally independent given $Z$. If instead we require $I(X,Y|Z)=I(X,Y)$, what do we get? What ...
2
votes
1answer
136 views

Relationship between least-squares regression and information theory

Is there a well-known relationship between least-squares regression and information theory? I've just started reading about information theory. It seems almost trivial to say that the regression ...
0
votes
1answer
27 views

KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0. What are the benefits of this? Is ...
0
votes
1answer
36 views

When is the conditional differential entropy, $h(X+Z_1\mid X+Z_2)$, maximized?

Let $Z_1$ & $Z_2$ be 2 i.i.d. RVs, each distributed according to $N(0,1)$, and let $X$ be an arbitrary RV with unit variance. What distribution of X will maximize this conditional differential ...
4
votes
1answer
93 views

Information gain and mutual information: different or equal?

I'm very confused about the difference between Information gain and mutual information. to make it even more confusing is that I can find both sources defining them as identical and other which ...
4
votes
2answers
501 views

Why would perfectly similar data have 0 mutual information?

I'm not a statistic major, so my knowledge of statistics is quite limited but I've found myself in need of learning about and using mutual information. I believe I understand the concept and formula, ...
2
votes
2answers
79 views

How “interesting” a data series is

I have a large dataset containing several objects. Each object has many attributes which is arranged in a time series. Is there a suggested method to find the top n "most interesting" attributes? The ...
9
votes
1answer
194 views

Is differential entropy always less than infinity?

For an arbitrary continuous random variable, say $X$, is its differential entropy always less than $\infty$? (It's ok if it's $-\infty$.) If not, what's the necessary and sufficient condition for it ...
2
votes
1answer
57 views

Shannon entropy and inequality of expectations

Consider two distinct probability distributions $P(X)$ and $Q(Y)$---defined on the same domain---with (Shannon) entropy of $H(X)$ and $H(Y)$. I am interested to prove that $$ H(X) \leq H(Y) \implies ...
2
votes
0answers
106 views

Numerical Approxmation of standard errors for parameter estimation in the EM algorithm

Generally, when you want to compute standard errors for estimated parameters within the ML framework, one uses the diagonal elements of the observed information matrix. In for instance ...
1
vote
0answers
21 views

lower bound on conditional differential entropy

I was wondering if there exists a lower bound of conditional differential entropy of Gaussian random variables. Formally, let $\mathcal{X}=\{X_1,\ldots,X_m\}$ and $\mathcal{Y}=\{Y_1,\ldots,Y_n\}$ ...
1
vote
0answers
21 views

How to aproach a presentation of information theory without solid background

I study first year of an applied statistics bachelor, in one of my courses i have to do a presentation of information theory and i would like to share with you my ideas about it in order that you ...
2
votes
1answer
28 views

Information content of a set of random variables

Suppose there is some distribution $F$ not known to us. However, we can get information about this distribution by means of samples, i.e. we have a set of random variables from this distribution. ...
0
votes
0answers
50 views

Information gain of flipping coins

I'm studying for an exam on Bishop's Pattern Recognition and Machine Learning (ISBN: 978- 0387310732). One of the questions in the mock exam paper is: Three fair coins are flipped sequentially; you ...
1
vote
0answers
36 views

entropy-calculation in R [closed]

Is there anyone to have used the package "entropy" ? How can I create a vector of counts if my sample is large (2000 points)? I tried creating a vector of counts using table(), is there a more ...
0
votes
0answers
73 views

Variable selection via mutual information

I am building a SVM model and was wondering whether filtering variables based on mutual information between response variable(binary 0/1 variable)is a good approach? Also, would the condinformation ...
2
votes
1answer
72 views

Information criterion for selecting sample size when modeling tails

I want to model the left tail of an unknown distribution with a Generalized Pareto distribution. Somehow I have to select how much of the tail to model. I am wondering if it is possible to create an ...
3
votes
0answers
32 views

Expected ratio of probabilities--is there a term for it?

I recently came across the following quantity when I played around with some information theoretic quantities and Bayesian learning. Given three probability distributions $q(z), p(z)$ and $p(z|x)$. ...
0
votes
1answer
52 views

How much prediction accuracy of SVM (or other ML models) depend on the way features are encoded?

Suppose that for a given ML problem, we have a feature which car the person possesses. We can encode this information in one of the following ways: Assign an id to each of the car. Make a column ...
2
votes
0answers
129 views

Use of information theory in applied data science

Today I ran across the book "Information theory: A tutorial introduction" by James Stone and thought for a moment or two about the extent of use of information theory in applied data science (if ...
0
votes
2answers
80 views

Clean data and noisy data: which one has higher entropy?

I have a question regarding to the information theory. Between clean data and noisy data, which one has higher entropy? I think the noisy data has, am I right? But, noisy data does not have more ...
0
votes
1answer
77 views

Transfer entropy on real, continuous data

Here is the goal and problem: I am trying to calculate a measure of coupling between real-valued, continuous oscillatory data. The data come from two people producing synchronized rhythmic ...
1
vote
1answer
268 views

How does the log(p(x,y)) normalize the pointwise mutual information?

I'm trying to understand the normalized form of pointwise mutual information. $npmi = \frac{pmi(x,y)}{log(p(x,y))}$ Why does the log joint probability normalize the pointwise mutual information to ...
2
votes
1answer
324 views

Calculating the mutual information between two histograms

I've been set a sample exercise by my supervisor, and I'm totally lost as to where I should be heading. What I've been tasked with is to generate two histograms that approximate Gaussian PDFs. Then, ...