Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

When is it difficult to estimate entropy and mutual information?

In what situations, or for what sort of data, is it challenging to estimate entropy, differential entropy and mutual information? or even infeasible? Is it hard only with small datasets and high ...
2
votes
1answer
25 views

What does maximizing mutual information do?

In information theory, there is something called the maximum entropy principle. Are other information measures, such as mutual information, also commonly maximized? If mutual information describes the ...
0
votes
0answers
11 views

How does the maximum entropy principle affect joint entropy, mutual information, and other info measures?

The maximum entropy principle says that we should use the probability distribution of a univariate dataset that has the highest level of entropy because it offers the lowest information. How does the ...
0
votes
0answers
27 views

Information Theoretical optimality of Jeffreys Prior?

Given a "conditional distribution" $f(x|\theta)$ and its corresponding Jeffreys prior $r(\theta)$, is there some information theoretic sense (in terms of quantities like mutual information) ...
3
votes
1answer
51 views

Interpretation of mutual information

Is it correct to say that the mutual information $I(X;Y)$ quantifies how well we can discriminate among the outcomes of $Y$ by looking at the outcomes of $X$ (or viceversa)? If yes, do you have any ...
0
votes
0answers
10 views

Social Network Analysis: assortativity between two classes and network structure

I have a network where nodes are labelled in two classes "a" and "b". I want to measure how connected these two groups are and I looked at assortativity by group. I want to use this measure in more ...
0
votes
1answer
41 views

Mutual information between a vector and a constant

Let's say that we have a tensor x sampled from its probability distribution, and that we have a constant vector c sampled from a degenerate distribution (its value is constant). Is the mutual ...
0
votes
0answers
27 views

Mutual information estimated on subsets of data

I am estimating the mutual information for a continuous data set using the kNN-based mutual information estimator proposed by Kraskov et al [1]. Lets consider two features $X$ and $Y$, and the ...
3
votes
0answers
30 views

Interpretable General Measure of Dependence

I am looking for an interpretable measure between two random variables $X$ and $Y$ which quantifies the dependence between the two but does not assume linearity. Essentially, I am looking for a ...
0
votes
0answers
15 views

Advice on anomaly detection for a text

I have an idea of retreiving texts that have anomalous distribution of token frequency. Let's say I have a corpus of texts, and I build a document-term matrix based on token frequency. Naturally, ...
1
vote
0answers
17 views

The monotonicity of entropy operator

Define the entropy operator of a distribution as $\mathbb{H}(p) = -\int p \log p$, how does the entropy change for distributions that are proportional to the powers of $p$? For example, define $\...
3
votes
1answer
58 views

Mutual information between $X$ and $f(X)$

Let $X$ and $Y$ be two random variables, where $Y=f(X)$ is a deterministic function of $X$. Furthermore suppose $X,Y$ are continuous and that $f$ is smooth. Is the mutual information between $X$ and $...
2
votes
2answers
40 views

Rewriting the mutual information of a linear model through conditional expectation

I am reading the following paper: http://web.mit.edu/18.325/www/telatar_capacity.pdf In this paper we have the following linear model, with $\mathbf{n}$ being additive noise: \begin{equation} \...
2
votes
0answers
24 views

Mutual Information as a Uncertainty Reduction Criteria For Arbitrary Field

I have been reviewing some papers such as this that uses Mutual Information (MI) as a criteria to obtain most informative point to approximate some large field an reduce uncertainty of the field. This ...
2
votes
1answer
60 views

Difference between Covariance and Mutual Information

I was going through the posts that describe the difference between covariance and MI and came across following from Quora The covariance of two random variables measures the strength of the linear ...
1
vote
1answer
39 views

Understanding Multidimensional Mutual Information

Given random variables $\vec{x}, \vec{y} \in \mathbb{R}^n$, and the mutual information, defined as $I(\vec{x} : \vec{y}) = H(\vec{x}) + H(\vec{y}) - H(\vec{x}, \vec{y})$ is it true that $I(\vec{x}: ...
0
votes
0answers
21 views

Mutual Information between covariates and response in linear regression

Suppose we obtain a linear regression model through least square minimization: $$ \hat y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 $$ Suppose $x_1 \sim N(0, \sigma_1^2)$, $x_2 \sim Ber(0.5)$. Is it ...
2
votes
1answer
83 views

Categorical variables sklearn random forest

I'm a bit confused with the use of Random Forest in Sklearn in case we have categorical variables. I've read this article stating that one hot encoding affects performance negatively when using ...
2
votes
0answers
54 views

Law of total expectation applied to conditional mutual information?

I came across a book where the author uses the following property of mutual information: Let $X$,$Y$,$Z$ be arbitrary discrete random variables and let $W$ be an indicator random variable. $$ (1)\ \ ...
0
votes
0answers
11 views

Characterization of two gaussian mixtures with common part

I'm looking for either the covariance, the joint entropy, something to ultimately get the mutual information $I(X,Y)$ where $X$ and $Y$ are defined as : $X = c*\epsilon_c + \epsilon_X$ $Y = c*\...
2
votes
1answer
68 views

“Any meaning to the concept of ‘Self Mutual Information?”

** “Any meaning to the concept of ‘Self Mutual Information?” ** A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
0
votes
1answer
42 views

Selecting Feature weights

I use the knn Classifier for a binary classification problem. To improve the classification results I would like to multiply features by weights that are learned from data. I found different ways to ...
1
vote
0answers
31 views

Estimate joint probability of two dependent variable

I've a dataset created in the following way: The input of the system is a 8 bit binary number ranging from $x_1$= 0000 0000 to $x_N$ = 1111 1111 For every input i've read the output of the system (...
0
votes
0answers
76 views

how to derive $I(x,x)=H(X)$

Trying to understand how to derive that the self mutual information I(X,X) is equal to the entropy H(X), as stated on the wiki page wiki mutual information, "therefore H(X) = I(X;X)". More than that, ...
0
votes
0answers
16 views

Simple worked-out example to compute mutual information between two random variables that are vectors

Here is a simple worked out example to compute MI between two random variables $X,Y\in \mathbb{R}$. This also does a good job. Suppose now i am dealing with two random vectors $P\in \mathbb{R}^3,\ Q\...
1
vote
0answers
45 views

Can linear (Pearson) correlation overestimate the true value?

It's well known that Pearson correlation can underestimate the true value in case of non-linear relationship. But can it overestimate? For example, Pearson correlation of discontinuous distributions, ...
3
votes
0answers
49 views

How to interpret the relationship between MI score and correlation score [closed]

I am confusing about the relationship between MI score and correlation score. I've searched for some information and know that: ...
0
votes
1answer
35 views

Mutual information between subsets of variables in the multivariate normal distribution

Let $\vec{X}$ be a random vector following a multi-variate normal distribution $P(\vec X)$ with covariance matrix $\Sigma$ and zero means (for simplicity). Consider a partition of $\vec X$ into two ...
2
votes
0answers
51 views

A mathematical definition of noteworthiness

If you're a sport fan, you've probably come across these bogus, cherry-picked stats, things like: "This is the first time a player has scored 32 points on the Ides of March when the temperature was ...
0
votes
0answers
19 views

Mutual Information for monotonic function mapping

Consider a random Gaussian input $\mathbf{x}\in\mathbb{R}^n \sim \mathcal{N}(0,q\mathbf{I})$ as a input to a deterministic function mapping as $y= f(<\mathbf{w,x}>) $ for a monotonic function f(...
0
votes
1answer
164 views

Feature selection : Mutual information between 2 features or between feature and target?

I am aware that Mutual Information (MI) is one solution for feature reduction techniques. Consider a binary model with feature vector X = (x1, x2,...,xn) and ...
1
vote
0answers
20 views

Understanding Mutual Information score between two variables

I have two variables - a target and a predictor. I want to capture the relationship between the two, but they are non-normal distributed, so I can't use correlation. I remember a Data Science ...
0
votes
0answers
18 views

measure of the difference between variation of two time series of probability distributions

I'm looking at a series of particle density probabilities of proteins floating on a cell, these particles move around and also blink, so these density probabilities differ a bit from one time to ...
0
votes
0answers
16 views

Proof help for multivariate mutual information as a sum of entropies

I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies. $J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(...
1
vote
0answers
13 views

Are there differentiable estimators for Entropy?

I have recently came across a paper on estimation of Information theoretic measure such as Entropy, Mutual Information and divergence, using a Mean Nearest Neighbor approach. Since, the estimator is ...
0
votes
0answers
61 views

Who to attribute information gain to?

I am writing a paper where I examine information gain specifically with regards to feature selection and am wondering what the proper reference should be. I have looked all over and I can't find a ...
1
vote
0answers
39 views

Which has more mutual information with a multivariate Gaussian: its first principal component, or its first factor?

I have a $k$-dimensional Gaussian random variable $X\sim\mathcal{N}(0, \Sigma_X)$. What I want is a 1-dimensional scalar r.v. $Y\sim\mathcal{N}(0,1)$ that is jointly Gaussian with $X$ while maximizing ...
1
vote
0answers
71 views

How to show an alternate data processing inequality concerning KL divergence between conditionals?

Suppose $(Y,X) \sim F \in \mathcal{P(\mathbb{R^d})}$. Consider an arbitrary transformation $f$ that acts on $X$. My intuition is that the following should be a result in information theory: $$ \...
0
votes
0answers
29 views

Finding PCA-like directions in feature space that maximise sensitivity to a target variable

I have a fairly large space of feature variables in which I want to build a predictor for a target variable. My input dataset for training the predictor are sampled from the space using a mix of log ...
3
votes
2answers
67 views

Using conditional mutual information with measurements

I want to compute the mutual information between two discrete random variables $x_1,x_2$ whose PMFs are affected by measurements resp. $z_1,z_2$ (one measurement per variable) in a known way. Based on ...
0
votes
1answer
39 views

Need help understanding Toru Tsujishita' theorem on Triple Information

I struggling to understand point (ii) of Toru Tsujishita's theorem (here) on Triple- (Interaction- or Co-) Information What is meant with the maps $\varphi_j$ and $\varphi_k$? A biunivocal ...
3
votes
1answer
287 views

Confusion about proof in “Representation Learning with Contrastive Predictive Coding”

In the Appendix A.1 of the paper "Representation Learning with Contrastive Predictive Coding", the author prove $\log N-\mathcal L_N$ is the lower bound of mutual information between $x_{t+k}...
0
votes
0answers
65 views

How to choose sample size from probability density for computing mutual information based on continuous variables

I need to compute mutual information gain based two continuous variables $X$ and $Y$ $I(X|Y) = \int_X\int_Y p_{x.y}(x,y) \log(\frac{p_{x.y}(x,y)}{p_{x}(x)p_{y}(y)})$. I have used Kernel Density ...
1
vote
1answer
55 views

If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
1
vote
0answers
131 views

Non-negativity of interaction information for special trivariate case

Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies $$ p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ), $$ for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
0
votes
1answer
42 views

Mutual Information between multi-dimensional and single dimensional Variables

I would like to estimate MI between two variables X and Y of shape (nXd) and ...
0
votes
1answer
197 views

Information Bottleneck principle of Deep Learning model implementation

I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other ...
0
votes
1answer
39 views

If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
0
votes
0answers
52 views

Sampling from conditional distribution given joint distribution

So in a problem I'm working with, I'm essentially given 200 (arbitrary number) of samples from a joint distribution $P(x, y, z)$. However, what I want to be able to do is to extract the mutual ...
0
votes
0answers
64 views

Choosing appropriate statistical tests

I am trying to run stats on medical treatment data and would like your guys and girls help Goal: Comparing two trends of annual percentages for statistical significance. Is one decreasing due to the ...

1
2 3 4 5 6