The mutual-information tag has no wiki summary.
0
votes
0answers
27 views
Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?
I need some metric of divergence of two distributions.
(They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right ...
0
votes
1answer
65 views
Modelling two correlated variables
I wish to simultaneously predict two correlated time series. Here is a plot of one time series against the other:
At the moment I have separate linear regressions for both of them which rely on ...
1
vote
0answers
13 views
Is there any meta-approach for variable selection based of measures of similarity between each two variables?
Is there any meta-approach ( or mayby I should say universal approach which works with different measures ) for variable selection which is based on similarity matrix which every entry ...
2
votes
2answers
122 views
Mutual Information based feature selection
Suppose I have a feature matrix $F = [f_1^T,f_2^T,...,f_m^T]$ whereby $f_j^T \in \mathbb R^{n_t \times 1}$ is the $j$th column vector of $F$ ($n_t$ is the number of different events/trials and $m$ is ...
0
votes
0answers
39 views
How to calculate entropy in log Scale?
I'm working on a problem , where a function returns the Log probability of P(X=x) . Now I would like to find the Entropy of X. But since the probabilities I get are log probabilities, again taking its ...
0
votes
0answers
34 views
Is there such a thing as conditional mutual information?
I have 2 random variables, A and B, both of which are categorical. I understand that I could calculate their mutual information, which would give me a measure of the information that they share. ...
1
vote
0answers
31 views
Is it possible to calculate mutual information by moments generating functions?
I went to listen to a workshop and some audience asked the presenter how the moments can improve the mutual information. I am learning the MI(Mutual Information) and moments so don't have enough ...
1
vote
0answers
22 views
Relationship between mutual information and the sign of the relationship
Assume we have two variables A and B, and we are trying to find the Mutual Information between them. Can the mutual information enable us to find if there exists a positive or negative relationship ...
1
vote
0answers
95 views
Mutual Information really invariant to invertible transformations?
"Estimating mutual information" [A Kraskov, H Stögbauer, P Grassberger - Physical Review E, 2004]
states that
Mutual information is invariant under reparametrization of the marginal variables. ...
0
votes
2answers
172 views
compare two time series
I have two time series that come from the same system. One is taken from the whole system, and the other os taken from, say, 10% of the system. The two time series have the same frequency. Is there a ...
1
vote
1answer
197 views
How to calculate mutual information?
I'am a bit confused. Can someone explain to me how to calculate mutual information betweet two terms based on a term-document matrix with binary term occurance as weights?
...
2
votes
0answers
67 views
Random sampling for estimating mutual information - Time complexity and sampling error?
I have a dataset and I want to compute the mutual information (MI) for a selected set of variables. The dataset is large enough so that computation of the MI may take undesirably long time. Can I just ...
0
votes
0answers
74 views
Is it possible to use Mic (maximal information coefficient) as an 'evolution' of partial correlation?
I have seen on a 11 months ago question here the following quote from a paper referred on the question:
...
2
votes
0answers
47 views
Multi-information of a uniformly distributed random variable on the L1 sphere
I posted this question in the stackexchange mathematics forum without any reponse. Maybe it was the wrong forum, so I try it here.
I tried to compute the multi-information (MI) $I[\mathbf U] = ...
0
votes
0answers
74 views
Conditional Entropy of linear combination of random variables
$X_1 \sim \mathcal{N}(0,P)$, $X_2 \sim \mathcal{N}(0,P)$ and $Z \sim \mathcal{N}(0,N)$ with $X_1,X_2,Z$ mutually independent.
How do we compute
$$
H(aX_1+bX_2+Z \mid \alpha X_1+\beta X_2)
$$
where ...
1
vote
0answers
49 views
How can I calculate the statistical power of a MI based likelihood ratio test with many df?
I'd like to calculate the statistical power of a likelihood ratio test based on mutual information.
I'm using the asymptotic property that the test statistic $G = 2NI$ where $N$ is the sample size ...
2
votes
0answers
154 views
KL divergence between 2 distributions with unequal cardinalities?
Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$.
Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or ...
1
vote
0answers
110 views
How do I calculate the p-value for each mutual information?
I've calculated the Mutual Information for my dataset using ARACNE, but I need to get p-values for each Mutual information. How do I calculate them? Is there any software for this calculation?
7
votes
2answers
501 views
Feature selection using mutual information in Matlab
I am trying to apply the idea of mutual information to feature selection, as described in these lecture notes (on page 5).
My platform is Matlab. One problem I find when computing mutual information ...
3
votes
0answers
79 views
Mutual information in a mixture of independent variables
Suppose we have a $n$ pairs of mutually independent variables over $k$ outcomes and take a mixture distribution, what can we say about mutual information in the mixture? In particular I'm wondering ...
2
votes
1answer
282 views
Question about calculating mutual information
I have two vectors $x,y \in \mathbb{R}.$ Based on the count and vector length I can compute $p(x)$ and $p(y)$ but I have no information on the joint density. How can I calculate mutual information in ...
2
votes
1answer
65 views
Inference on random graph, with an application to mobile sensors
I've attended a course on Machine Learning and another one in Network Analysis, and I wonder if this two topics already intersect, in particular I'm interested in the following model:
we have a ...
2
votes
0answers
99 views
Normalized mutual information using cardinalities
I have the cardinalities of sets $N_i, \forall i \in 1,2..n$, and the cardinalities $|N_i \cap T|, |N_i|, |T|, |N_i \cup T|, \forall i$, are known.
Here, the set $T$ and sets $N_i$'s are all ...
1
vote
2answers
87 views
Mutual information with a Dirac delta type pdf
What does the $MI(X,Y)$ convey about $Y$, when one of the probability distributions, $X$ is trivial and has all the probability concentrated at a single point?
1
vote
0answers
52 views
Tractability of mutual information-augmented ensemble classification algorithms
I am seeking to augment random forest classification using Shannon-Weaver mutual information as a metaheuristic to partition candidate datasets. Specifically, I am trying to determine if such an ...
2
votes
1answer
2k views
Estimating the mutual information for two signal samples
I have two signals s1 and s2, sampled 170 times each.
x = 0.2:0.2:34;
s1 = rand(size(x));
s2 = randn(size(x));
The calculation of the MI (mutual information) ...
7
votes
2answers
467 views
Cross correlation vs mutual information
What is the difference between Cross Correlation and Mutual Information. What kind of problems can be solved using these measures and when is it appropriate to use one over the other.
Thanks for the ...
4
votes
2answers
300 views
How to calculate threshold level for mutual information scores?
Im quite new to the world of statistics, and only just finished work on my first (successful) attempt to calculate mutual information scores.
Now lets say that for the column pairs AB, BC, CD and DE ...
3
votes
1answer
380 views
Can mutual information gain value be greater than 1
I have a very basic doubt. Sorry if this irritates few. I know that Mutual Information value should be greater than 0, but should it be less than 1 ? Is it bounded by any upper value ?
Thanks,
Amit.
5
votes
1answer
737 views
Distance correlation versus mutual information
I've worked with the mutual information for some time. But I found a very recent measure in the "correlation world" that can also be used to measure distribution independence, the so called "distance ...
1
vote
1answer
336 views
The disadvantage of using F-score in feature selection
F-score can be used to measure the discrimination of two sets of real-numbers and can be used for feature selection. However, I once read that
A disadvantage of F-score is that it does not reveal ...
14
votes
3answers
1k views
Can the MIC algorithm for detecting non-linear correlations be explained intuitively ?
More recently, I read two articles. First one is about the history of the correlation and second is about the new method called Maximal Information Coefficient (MIC). I need your help regarding to ...
3
votes
1answer
318 views
Calculating mutual Information
I am making a spam filter for an assignment. One of the steps outlined in a paper that I have read indicates finding the mutual information for all the words in the text and then choosing the 500 or ...
1
vote
0answers
125 views
Pointwise mutual information for tag generation in Twitter
I came across the following link. http://stackoverflow.com/questions/2764116/tag-generation-from-a-small-text-content-such-as-tweets
The question is about how do we assign tag to tweets ...
7
votes
1answer
252 views
Why do people use the term “weight of evidence” and how does it differ from “pointwise mutual information”?
Here, "weight of evidence" (WOE) is a common term in the published scientific and policy-making literature, most often seen in the context of risk assessment, defined by:
$$w(e : h) = ...
3
votes
1answer
83 views
High dimensional volume entropy estimator
I am writing a program using high-dimensional volume (HDV) estimator to estimate entropy and mutual information for variable selection. Let $ D = (x^i_1, x^i_2, ..., x^i_M)$, N is the number of data ...
1
vote
0answers
72 views
Is there a way to measure the information content (or difference) in two matrices?
Prologue
I have a matrix $N$ with dimensions $I\times J$, $I >> J$, and $n_{ij}$ is assumed to follow a Poisson distribution with mean $\lambda_{ij}$. Using a method (say NtoM), I reduce $N$ to ...
7
votes
1answer
251 views
Bounding mutual information given bounds on pointwise mutual information
Suppose I have two sets $X$ and $Y$ and a joint probability distribution over these sets $p(x,y)$. Let $p(x)$ and $p(y)$ denote the marginal distributions over $X$ and $Y$ respectively.
The mutual ...
4
votes
2answers
243 views
How can I compare the following mutual information values?
How can I compare the following mutual information values ? I'm just wondering what's the most appropriate way to display them in my report table.
I'm computing them with this formula
where e and c ...
5
votes
1answer
599 views
Pointwise mutual information for text using R
I have key-value pairs of text. The values can be multiple words (n-grams). For example,
A abcd
A efgh
B abcd
C wxyz
C mnop
I want to ...
1
vote
2answers
363 views
Can the mutual information of a “cell” be negative?
Please forgive me if this is not the right Stack Exchange (and for inventing terms).
For discrete random variables X and Y, the mutual information of X and Y can be defined as follows:
$I(X;Y) = ...
1
vote
1answer
530 views
What are the advantages of using mutual information over Pearson correlation in network inference?
Does mutual information discriminate against fold change differences?
1
vote
3answers
414 views
Is mutual information invariant to scaling, i.e. multiplying all elements by a nonzero constant?
I am trying to compare it to Euclidean distance and Pearson correlation
9
votes
2answers
2k views
What is the major difference between correlation and mutual information?
my question particularly applies to network reconstruction
