# Questions tagged [information-theory]

A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in predicting a random variable.

368 questions
13 views

### Connections between logistic regression, information value and Kullback-Leibler

Suppose that we are interested in modeling a binary predictor $Y=0,1$ subject to $m$ predictors $x_1,...,x_m$. First, let us examine a simpler model of the impact of $x_j$ on the response $Y$. By the ...
20 views

### Am I using conditional entropy formula correctly?

I want to use the formula for a measure of complexity of a system: $$C(X) = H(X) - \sum_{x \in X}{H(x\mid X-x)}$$ where $x$ is a subpart of the system $X,$ and $H(X)$ is shannon entropy. Then I have ...
14 views
+100

### How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
13 views

### Can the information entropy limit be achieved for any distribution that contains at least one probability mass that is not a unit fraction?

In a symbol-by-symbol coding with a known probability distribution, such as the Huffman coding, can the entropy limit be achieved in any case where there's a data value with a probability that's not ...
13 views

### Analytical expression of the minimizer of cross entropy loss when the predicted function is a constant fucntion?

Let $\{y_1...y_n\} \in \{0,1\}$, and let $c \in [0,1]$. Define the cross-entropy of loss of $c$ by: $$C(c): = \sum_{j=1}^{n}- y_j ln c - (1- y_j) ln (1-c)$$. Define $c*= arg min _{c} C(c)$ Is ...
43 views
+50

### Combining subjoint distributions to create a larger joint distribution

I am trying to construct large joint distributions through smaller joint distributions and I'm not sure how to approach the literature. I am curious if there exists a function which can take n ...
16 views

### Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
20 views

### Feature selection via conditional entropy

It looks like feature selection can be done with mutual information. Mutual information is related to conditional entropy by this equation: $I(X,Y) = H(X) - H(X|Y)$ Can we use conditional entropy ...
14 views

### Conditional entropy of an outcome

Given three discrete random variables $X$, $Y$ and $Z$, the conditional entropy $$H(X|Y,Z) = \sum_{X}\sum_{Y}\sum_{Z} p(x,y,z) \ \text{log}\frac{p(y,z)}{p(x,y,z)}$$ If I want to calculate the ...
38 views

### Convexity of cross entropy

I am not sure if this is a better fit for this site or mathematics.stackexchange but I've seen similar questions on here before. I'd like to know if the following is true and if so, how I could go ...
51 views

### Can information entropy be computed on arbitrary set of non-negative numbers smaller than 1?

I have a set of non-negative numbers smaller than 1 which do not sum to 1. These numbers will be used in some weighted sum. I'm wondering in this case can we still measure the degree of concentration ...
20 views

### Why H(y) is a concave function of p(x) when p(y) is a linear function of p(x)?

I got stuck in the line when reading a theorem in The Elements of Information Theory on page 59: If p(y|x) is fixed, then p(y) is a linear function of p(x). Hence H(Y), which is a concave ...
32 views

### How to check if covariates in multiple regression is explaining the same?

I am a master's student doing my thesis at the moment and have come to the point of determining my empirical setup. I would like to get some guidance, in terms of what I am thinking is proper.. I ...
79 views

### Interpretation of spectral entropy of a timeseries

The tsfeatures package for R has an entropy() function. The vignette for the package describes it as: The spectral entropy is ...
29 views

### Measure of randomness that maps well to “hard to guess”?

It was recently pointed out to me that entropy does not necessarily map well to "hard to guess". E.g.: consider a distribution where there's a very high (say, 90%) chance you select a single, ...
33 views

### Uniform distribution on the simplex. - Thomas cover

I'm trying to formulate the solution for the following problem: I was thinking in finding the equivalent distribution on $X_i$ based on $Y_i$, but I think I'm cheating. I think that the autor wants ...
19 views

### Keeping starting variables after a Principal component analysis

I know the PCA theory and how keeping the k-first principal components help dimensionality reduction. But what about keeping the base vectors ? You start with $n$ row vectors $x_1,...,x_n$, ...
31 views

### Which properties yield the exponential family of distributions?

It seems like every resource that discusses exponential families simply defines the family of distributions, explains why it's useful and then derives some of its properties. I have only seen one ...
18 views

### joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
18 views

### What does Joint entropy really tells us?

I have two sequences of data A and B. A = [sunny, sunny, cloudy, rainy, sunny, sunny] with entropy 1.25 bits b = [hot, hot, cold, cold, cold, hot] with entropy 1 bits I have calculated joint ...
17 views

### Multivariate Conditional Entropy as a test of correlation between random variables

I use the word columns to mean the data from which a random variable can be estimated. It is a sample of a random variable. I am working with $N$ columns of weakly correlated data. Furthermore, I ...
33 views

### Estimating Differential entropy from unbiased samples of a probability distribution

If I can get N unbiased samples $x$ from $p(x)$ how can I approximate the Differential entropy: $$H(X) = -\displaystyle\int_{x} p(x)\log p(x) dx$$ I'm not very knowledgeable in statistics so I'm not ...
29 views

### SHAP values vs Information Gain?

SHAP values which are essentially the variable importance at a local level where each variable's importance is assessed by different in probability outcome of a model with and without the variable. I ...
11 views

### Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
24 views

### Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
19 views

### Information gain or regression for measuring to what extent a continuous quantity detemines a discrete one?

I have a small population S and two quantities P and Q associated to its members. P is integer valued (say, age taking integer values between 1 and 20) and Q real valued (say, height in meters). I can ...
9 views

### How to assess the informativeness of sets of past observations?

I know how to forecast one timeseries given its own past values and the values of other timeseries. However, I'm currently having a problem where the past values of a timeseries are partitioned into ...
17 views

### Prior/degree of belief/degree of lack-of-information/algorithms/complexity

For a long time I had a bit of difficulty understanding what "degree of belief" means. Recently I had some thoughts about it and I wonder if they make any sense, or is there some literature about ...
102 views

### Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
15 views

### Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
86 views

17 views

### What happen to gain ratio when information gain is 0?

I am learning decision tree using C4.5, stumbled across data where its attributes has only one value, because of only one value, when calculating the information gain it resulted with 0. Because ...
35 views

### Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
38 views

### What Is Meant by “Maximising” Posterior Probability?

My textbook says the following: The optimal coding decision (optimal in the sense of having the smallest probability of being wrong) is to find which value of $\mathbf{s}$ is most probable, ...
33 views

### Aikaike Information Criterion: derivation in original paper

I have been reading AIC paper 'Information theory and an extension of the maximum likelihood principle' by Akaike (1974). I have been able to understand up to the third section of the paper, but I am ...
33 views

### How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
58 views

### Can we apply KL divergence to the probability distributions on different domains?

When I was reading the original paper of t-SNE, I had an question whether or not we can apply KL divergence to the discrete probability distributions on different domains. In the paper, they measure ...
305 views

### What is the actual significance of a difference in AIC or BIC values?

Usually, when a difference of a statistic is discussed, that discussion is presented in the context of a significance of that difference. When self-entropy, i.e., information content, is examined, ...
10 views

### Can we calculate the best possible test set loss given the information in the training set?

Given a training set and a test set for machine learning, I am wondering if I can somehow measure if the information to predict the labels of the test set correctly is present in the training set. In ...
64 views

### reading books about Bayesian Model selection

I was trying to find some "good" reading books about Bayesian Model selection. So is there any recommendations? To be specific, I was trying to understand the Bayesian Information Criterion (BIC), the ...
254 views

### Correcting Kullback-Leibler divergence for size of datasets

We have the following implementation of KLD: ...
20 views

### Estimating a surprise of a word in context

What will be the best way to estimate the entropy/surprise of a word in a specific context? Let's say to compare the surprise of: context: "I watched the movie in my" word: Computer I ...
73 views

### Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...
49 views

### Why maximum likelihood estimation is same with minimizing cross-entropy? [duplicate]

Lots of articles say that MLE is as same as minimizing cross-entropy. I tried to prove this but failed. the relationship between maximizing the likelihood and minimizing the cross-entropy This ...
What is the VC dimension of $k$ finite unions of one-sided intervals: If we take 3 one-sided intervals like $(-\infty, a_1]$, $(-\infty, a_2]$ and $(-\infty, a_3]$, I think union of these ...