Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Make your voice heard. Take the 2019 Developer Survey now

Questions tagged [information-theory]

A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in predicting a random variable.

0
votes
0answers
10 views

Multivariate Conditional Entropy as a test of correlation between random variables

I use the word columns to mean the data from which a random variable can be estimated. It is a sample of a random variable. I am working with $N$ columns of weakly correlated data. Furthermore, I ...
2
votes
2answers
25 views

Estimating Differential entropy from unbiased samples of a probability distribution

If I can get N unbiased samples $x$ from $p(x)$ how can I approximate the Differential entropy: $$H(X) = -\displaystyle\int_{x} p(x)\log p(x) dx$$ I'm not very knowledgeable in statistics so I'm not ...
0
votes
0answers
6 views

SHAP values vs Information Gain?

SHAP values which are essentially the variable importance at a local level where each variable's importance is assessed by different in probability outcome of a model with and without the variable. I ...
0
votes
0answers
10 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
0
votes
0answers
21 views

Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
0
votes
0answers
17 views

Information gain or regression for measuring to what extent a continuous quantity detemines a discrete one?

I have a small population S and two quantities P and Q associated to its members. P is integer valued (say, age taking integer values between 1 and 20) and Q real valued (say, height in meters). I can ...
0
votes
0answers
9 views

How to assess the informativeness of sets of past observations?

I know how to forecast one timeseries given its own past values and the values of other timeseries. However, I'm currently having a problem where the past values of a timeseries are partitioned into ...
0
votes
1answer
17 views

Prior/degree of belief/degree of lack-of-information/algorithms/complexity

For a long time I had a bit of difficulty understanding what "degree of belief" means. Recently I had some thoughts about it and I wonder if they make any sense, or is there some literature about ...
1
vote
2answers
92 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
1
vote
0answers
12 views

Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
1
vote
1answer
63 views

Finding the MVUE from two independent random samples

Suppose we have a random sample $X_1, X_2, \ldots, X_n$ from exponential$~(β >0)$ $\text{i.e. }f(x\mid β) = {1/β} ~e^{−x/β}$ and a random sample$~Y_1, Y_2, \ldots, Y_n$ from exponential$~(⍺ >0)...
1
vote
1answer
48 views

Multivariate conditional entropy

I would like to take data columns and compute the multivariate conditional entropy. For instance, suppose I have columns $A, B, C D, E$ and I want to compute the conditional entropy $H(E | A,B,C,D)$. ...
0
votes
0answers
12 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
18 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
0
votes
0answers
14 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
12 views

What happen to gain ratio when information gain is 0?

I am learning decision tree using C4.5, stumbled across data where its attributes has only one value, because of only one value, when calculating the information gain it resulted with 0. Because ...
2
votes
1answer
25 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
-1
votes
0answers
8 views

How to measure word similarity using wordnet for the information content definition as detailed in Resnik 1995?

Resnik 1995 equation 3 uses count(n) to define P(c). What is count(n)? Any solved example on actual calculation would be much appreciated. Please move the question to relevant site if this doesn't ...
0
votes
1answer
30 views

What Is Meant by “Maximising” Posterior Probability?

My textbook says the following: The optimal coding decision (optimal in the sense of having the smallest probability of being wrong) is to find which value of $\mathbf{s}$ is most probable, ...
0
votes
0answers
31 views

Aikaike Information Criterion: derivation in original paper

I have been reading AIC paper 'Information theory and an extension of the maximum likelihood principle' by Akaike (1974). I have been able to understand up to the third section of the paper, but I am ...
1
vote
0answers
24 views

How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
2
votes
1answer
36 views

Can we apply KL divergence to the probability distributions on different domains?

When I was reading the original paper of t-SNE, I had an question whether or not we can apply KL divergence to the discrete probability distributions on different domains. In the paper, they measure ...
4
votes
2answers
273 views

What is the actual significance of a difference in AIC or BIC values?

Usually, when a difference of a statistic is discussed, that discussion is presented in the context of a significance of that difference. When self-entropy, i.e., information content, is examined, ...
0
votes
0answers
10 views

Can we calculate the best possible test set loss given the information in the training set?

Given a training set and a test set for machine learning, I am wondering if I can somehow measure if the information to predict the labels of the test set correctly is present in the training set. In ...
2
votes
1answer
57 views

reading books about Bayesian Model selection

I was trying to find some "good" reading books about Bayesian Model selection. So is there any recommendations? To be specific, I was trying to understand the Bayesian Information Criterion (BIC), the ...
3
votes
2answers
174 views
0
votes
0answers
20 views

Estimating a surprise of a word in context

What will be the best way to estimate the entropy/surprise of a word in a specific context? Let's say to compare the surprise of: context: "I watched the movie in my" word: Computer I ...
4
votes
0answers
71 views

Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...
1
vote
0answers
48 views

Why maximum likelihood estimation is same with minimizing cross-entropy? [duplicate]

Lots of articles say that MLE is as same as minimizing cross-entropy. I tried to prove this but failed. the relationship between maximizing the likelihood and minimizing the cross-entropy This ...
1
vote
0answers
20 views

How Much More Information (as bits) is Gained (if any) from a Ranking vs. Likert Scale Survey of 100 Questions's / 1-5 Scale?

All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a ...
0
votes
0answers
80 views

VC dimension of finite unions of one-sided intervals

What is the VC dimension of $k$ finite unions of one-sided intervals: If we take 3 one-sided intervals like $(-\infty, a_1] $, $(-\infty, a_2] $ and $(-\infty, a_3] $, I think union of these ...
2
votes
2answers
445 views

Random Forests and Information gain

Suppose you are building random forest model, which split a node on the attribute, that has highest information gain. In the below image, select the attribute which has the highest information gain? ...
1
vote
2answers
55 views

At what sample size does the precision of estimate stabilise?

Imagine I have a population of a known size (e.g., 100,000 voters), and I know they can only vote for A, B, C, D or E. If a random sample of 100 voters are, e.g., A = 40%, B = 30%, C = 10%, D = 10%, ...
0
votes
1answer
56 views

Is there a metric of representativeness in statists?

In a group of 1000 people, where each of them is either A, B, C, D or E (no one can be more than one thing). If I have exactly 200 people with each trait (20% per trait), when I take a random sample ...
0
votes
0answers
35 views

Deterministic limit of “ML” and it's generalization to the non-deterministic case

I was told a few years ago at school (grad school) that for independent features the Naive Bayes classifier is very good, "almost perfect". Especially if we consider binary features because then the ...
1
vote
1answer
182 views

Kullback-Leibler divergence is not symmetric but why mutual information is symmetric?

As we know Kullback-Leibler divergence is not symmetric and mutual information is the KL divergence between P(X,Y) and P(X)*P(Y).So my question is that why mutual information is symmetric which is a ...
0
votes
2answers
88 views

Entropy or co-occurence matrix to compute the randomness of gray scale images?

I hope this is the right place to ask this question. I have an algorithm that outputs gray scale images (not normalized). These images oftentimes contain a lot of random noise and sometimes ...
0
votes
1answer
85 views

Computing joint entropy from marginal distributions

I have distributions of N random variables (supposed conditionally independent) consequently, the joint distribution is the multiplication of all the distributions. I want to compute the joint ...
2
votes
0answers
146 views

Can the differential entropy be negative infinity?

Define the (differential) entropy for density $f$ as $$ H(f) :=-\int_{0}^{1} f(x) \log_{2}(f(x)) dx \, .$$ I am trying to find a Lebesgue measurable $f$ defined on $[0,1]$ such that $f\geq 0, \...
0
votes
2answers
50 views

Looking for a method to assess the quality of a dataset for a specific task

Here I have a question that might seem crazy, but I appreciate any thoughts and experiences: When we have a dataset and a classification task at hand, we usually look for a Machine Learning (or in ...
0
votes
0answers
34 views

Entropy is a measure on distributions, mutual information is a measure on RVs

In a lecture I read this statement, no explanation was given "Unlike entropy, mutual information is a measure on random variables, not on distributions" Can anyone explain what this means? From ...
0
votes
1answer
29 views

Shannon entropy with regards to independent random variables

I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific ...
1
vote
0answers
27 views

Why do we care about maximum entropy? [duplicate]

One justification for the ubiquity of the (multivariate) normal distribution in statistical/machine learning modeling is that it maximizes entropy among distributions with mean $\mu$ and variance ...
2
votes
0answers
46 views

What is the intuition regarding the different viewpoints of the Kullback-Leibler Divergence and the Kolmogorov-Smirnov Statistic? [duplicate]

In trying to understand the Kullback-Leibler Divergence I conceive it as a metric that if minimized would make the Approximation PDF Q as close as possible to the True PDF P either in the ...
1
vote
0answers
20 views

Conditions Mutual Information and Confounding Effect

Given that conditional mutual information (CMI) I(A,B |C) is the information shared between A, and B given C, does this consider the confounding effect -if any - that C introduces? In other words, ...
7
votes
2answers
170 views

Interpretation of entropy for continuous distribution?

"Entropy" roughly captures the degree of "information" in a probability distribution. For discrete distributions there is a far more exact interpretation: The entropy of a discrete random variable ...
1
vote
1answer
462 views

How to compute joint entropy of high-dimensional data?

Normally, I compute the (empirical) joint entropy of some data, using the following code: ...
1
vote
0answers
239 views

Trying to implement the Jensen-Shannon Divergence for Multivariate Gaussians

Given two multivariate Gaussian distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them. ...
2
votes
0answers
152 views

If the AIC is an estimate of Kullback-Leibler Divergence, then why can AIC be negative when KL divergence is always positive?

I have read many times that the AIC serves as an estimate of the KL divergence, and I know that AIC can be a negative value (and have seen that myself). Yet, the KL divergence must always be positive. ...
2
votes
1answer
107 views

Entropy of the beta-binomial compound distribution

I have a generative process as follows: $$ x \mid \alpha \sim \textsf{Beta}\left (\alpha,\beta \right) \\ y \mid x \sim \textsf{Bernoulli}(x). $$ How does one go about calculating the Entropy of ...