# Tagged Questions

The tag has no wiki summary.

24 views

### Mutual Information vs ANOVA

In the context of analysing dependencies between a continuous and categorical variables, what is the difference between mutual information and an ANOVA test with post hoc analysis? I'm aware that ...
29 views

### Information theoretic substitute for precision and recall?

Is there a substitute for measuring precision, recall of a classifier (binary or multi-class) to evaluate its performance using information-theoretic quantities like entropy, mutual information or ...
57 views

### conditional mutual information and how to deal with zero probabilities

the conditional mutual between three sets of mutually exclusive variables, X, Y, and Z, is defined as follows. $I(X,Y|Z) = \sum_{xyz} P(x,y,z) \log \frac{P(z)P(x,y,z)}{P(x,z)P(y,z)}$ my questions ...
16 views

### Bayesian Classifier and Mutual Information

http://en.wikipedia.org/wiki/Mutual_information Question. In Naive Bayesian Classification, is it better or more accurate to have less feature words? I built as an independent project to analyze ...
15 views

### cluster - class dependency bag of words [duplicate]

I have three clusters which is obtained by clustering features from images. Lets assume I have two classes. Consider the following table, where c denotes the class number and k denotes the cluster ...
84 views

### Computing mutual information

I have a problem when computing the mutual information between two variables. Let's consider the following table: ...
37 views

### Mutual information/pointwise mutual information for measuring prediction

I want to measure how well I predict a vector $Y$ (vector not a label) for observation $X$. Both $X$ and $Y$ have the same set of features ($1\times n$). For that, I thought of "scoring" the ...
152 views

### Using Mutual Information for Binary Logistic Regression Variable Selection

In addition to proc varclus, randomForest, and assessing multicollinearity among potential predictor variables, I am seeking ...
182 views

### Why don't statisticians use mutual information as a measure of association?

I've seen a couple talks by non-statisticians where they seem to reinvent correlation measures using mutual information rather than regression (or equivalent/closely-related statistical tests). I ...
48 views

### Relations between probabilities of “almost” independent random variables

Let $X$ and $Y$ be two random variables, such that the (average) mutual information is very small: $$0 \le I(X;Y) \le \epsilon \ll 1$$ In this case, we say that $X$ and $Y$ are almost independent. ...
91 views

### Entropy of (Sum of Gaussians) versus Sum of (Entropy of Gaussians)

Short version: How can the joint entropy of two independent variables be less than the sum of those independent variables? The joint entropy should encode all information that a scalar function can, ...
73 views

### How to combine the information of two probabilities?

I have a segmentation problem that using two different models I can assign a probability to each pixel of the image to be a background part. Two models produce slightly different results. Some pixel ...
49 views

### Mutual Information for clustering

I'm working on a document clustering application and decided to use Normalized Mutual Information as one of the measures of effectivenes. But I don't really understand how to implement this in that ...
28 views

### Testing if conditional mutual information is not 0?

I need to test if the conditional mutual information between two variables is NOT 0. While the null hypothesis is obviously that the conditional mutual information is 0, I can't think of an existing ...
80 views

### Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?

I need some metric of divergence of two distributions. (They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right ...
83 views

### Modelling two correlated variables

I wish to simultaneously predict two correlated time series. Here is a plot of one time series against the other: At the moment I have separate linear regressions for both of them which rely on ...
17 views

### Is there any meta-approach for variable selection based of measures of similarity between each two variables?

Is there any meta-approach ( or mayby I should say universal approach which works with different measures ) for variable selection which is based on similarity matrix which every entry ...
515 views

### Mutual Information based feature selection

Suppose I have a feature matrix $F = [f_1^T,f_2^T,...,f_m^T]$ whereby $f_j^T \in \mathbb R^{n_t \times 1}$ is the $j$th column vector of $F$ ($n_t$ is the number of different events/trials and $m$ is ...
69 views

### How to calculate entropy in log Scale?

I'm working on a problem , where a function returns the Log probability of P(X=x) . Now I would like to find the Entropy of X. But since the probabilities I get are log probabilities, again taking its ...
65 views

### Is there such a thing as conditional mutual information?

I have 2 random variables, A and B, both of which are categorical. I understand that I could calculate their mutual information, which would give me a measure of the information that they share. ...
46 views

### Is it possible to calculate mutual information by moments generating functions?

I went to listen to a workshop and some audience asked the presenter how the moments can improve the mutual information. I am learning the MI(Mutual Information) and moments so don't have enough ...
26 views

### Relationship between mutual information and the sign of the relationship

Assume we have two variables A and B, and we are trying to find the Mutual Information between them. Can the mutual information enable us to find if there exists a positive or negative relationship ...
143 views

### Mutual Information really invariant to invertible transformations?

"Estimating mutual information" [A Kraskov, H StÃ¶gbauer, P Grassberger - Physical Review E, 2004] states that Mutual information is invariant under reparametrization of the marginal variables. ...
218 views

### Compare two time series

I have two time series that come from the same system. One is taken from the whole system, and the other is taken from, say, 10% of the system. The two time series have the same frequency. Is there a ...
330 views

### How to calculate mutual information?

I'am a bit confused. Can someone explain to me how to calculate mutual information betweet two terms based on a term-document matrix with binary term occurance as weights? ...