Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [information-theory]

A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in predicting a random variable.

0
votes
0answers
13 views

Connections between logistic regression, information value and Kullback-Leibler

Suppose that we are interested in modeling a binary predictor $Y=0,1$ subject to $m$ predictors $x_1,...,x_m$. First, let us examine a simpler model of the impact of $x_j$ on the response $Y$. By the ...
0
votes
0answers
20 views

Am I using conditional entropy formula correctly?

I want to use the formula for a measure of complexity of a system: $$C(X) = H(X) - \sum_{x \in X}{H(x\mid X-x)}$$ where $x$ is a subpart of the system $X,$ and $H(X)$ is shannon entropy. Then I have ...
1
vote
0answers
14 views
+100

How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
0
votes
0answers
13 views

Can the information entropy limit be achieved for any distribution that contains at least one probability mass that is not a unit fraction?

In a symbol-by-symbol coding with a known probability distribution, such as the Huffman coding, can the entropy limit be achieved in any case where there's a data value with a probability that's not ...
0
votes
1answer
13 views

Analytical expression of the minimizer of cross entropy loss when the predicted function is a constant fucntion?

Let $\{y_1...y_n\} \in \{0,1\}$, and let $c \in [0,1]$. Define the cross-entropy of loss of $c$ by: $$C(c): = \sum_{j=1}^{n}- y_j ln c - (1- y_j) ln (1-c) $$. Define $c*= arg min _{c} C(c)$ Is ...
1
vote
0answers
43 views
+50

Combining subjoint distributions to create a larger joint distribution

I am trying to construct large joint distributions through smaller joint distributions and I'm not sure how to approach the literature. I am curious if there exists a function which can take n ...
0
votes
0answers
16 views

Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
0
votes
1answer
20 views

Feature selection via conditional entropy

It looks like feature selection can be done with mutual information. Mutual information is related to conditional entropy by this equation: $I(X,Y) = H(X) - H(X|Y)$ Can we use conditional entropy ...
0
votes
0answers
14 views

Conditional entropy of an outcome

Given three discrete random variables $X$, $Y$ and $Z$, the conditional entropy $$ H(X|Y,Z) = \sum_{X}\sum_{Y}\sum_{Z} p(x,y,z) \ \text{log}\frac{p(y,z)}{p(x,y,z)} $$ If I want to calculate the ...
1
vote
2answers
38 views

Convexity of cross entropy

I am not sure if this is a better fit for this site or mathematics.stackexchange but I've seen similar questions on here before. I'd like to know if the following is true and if so, how I could go ...
0
votes
0answers
51 views

Can information entropy be computed on arbitrary set of non-negative numbers smaller than 1?

I have a set of non-negative numbers smaller than 1 which do not sum to 1. These numbers will be used in some weighted sum. I'm wondering in this case can we still measure the degree of concentration ...
0
votes
0answers
20 views

Why H(y) is a concave function of p(x) when p(y) is a linear function of p(x)?

I got stuck in the line when reading a theorem in The Elements of Information Theory on page 59: If p(y|x) is fixed, then p(y) is a linear function of p(x). Hence H(Y), which is a concave ...
0
votes
1answer
32 views

How to check if covariates in multiple regression is explaining the same?

I am a master's student doing my thesis at the moment and have come to the point of determining my empirical setup. I would like to get some guidance, in terms of what I am thinking is proper.. I ...
2
votes
1answer
79 views

Interpretation of spectral entropy of a timeseries

The tsfeatures package for R has an entropy() function. The vignette for the package describes it as: The spectral entropy is ...
2
votes
1answer
29 views

Measure of randomness that maps well to “hard to guess”?

It was recently pointed out to me that entropy does not necessarily map well to "hard to guess". E.g.: consider a distribution where there's a very high (say, 90%) chance you select a single, ...
4
votes
0answers
33 views

Uniform distribution on the simplex. - Thomas cover

I'm trying to formulate the solution for the following problem: I was thinking in finding the equivalent distribution on $X_i$ based on $Y_i$, but I think I'm cheating. I think that the autor wants ...
0
votes
0answers
19 views

Keeping starting variables after a Principal component analysis

I know the PCA theory and how keeping the k-first principal components help dimensionality reduction. But what about keeping the base vectors ? You start with $n$ row vectors $x_1,...,x_n$, ...
3
votes
0answers
31 views

Which properties yield the exponential family of distributions?

It seems like every resource that discusses exponential families simply defines the family of distributions, explains why it's useful and then derives some of its properties. I have only seen one ...
0
votes
0answers
18 views

joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
0
votes
0answers
18 views

What does Joint entropy really tells us?

I have two sequences of data A and B. A = [sunny, sunny, cloudy, rainy, sunny, sunny] with entropy 1.25 bits b = [hot, hot, cold, cold, cold, hot] with entropy 1 bits I have calculated joint ...
0
votes
0answers
17 views

Multivariate Conditional Entropy as a test of correlation between random variables

I use the word columns to mean the data from which a random variable can be estimated. It is a sample of a random variable. I am working with $N$ columns of weakly correlated data. Furthermore, I ...
2
votes
2answers
33 views

Estimating Differential entropy from unbiased samples of a probability distribution

If I can get N unbiased samples $x$ from $p(x)$ how can I approximate the Differential entropy: $$H(X) = -\displaystyle\int_{x} p(x)\log p(x) dx$$ I'm not very knowledgeable in statistics so I'm not ...
0
votes
0answers
29 views

SHAP values vs Information Gain?

SHAP values which are essentially the variable importance at a local level where each variable's importance is assessed by different in probability outcome of a model with and without the variable. I ...
0
votes
0answers
11 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
0
votes
0answers
24 views

Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
0
votes
0answers
19 views

Information gain or regression for measuring to what extent a continuous quantity detemines a discrete one?

I have a small population S and two quantities P and Q associated to its members. P is integer valued (say, age taking integer values between 1 and 20) and Q real valued (say, height in meters). I can ...
0
votes
0answers
9 views

How to assess the informativeness of sets of past observations?

I know how to forecast one timeseries given its own past values and the values of other timeseries. However, I'm currently having a problem where the past values of a timeseries are partitioned into ...
0
votes
1answer
17 views

Prior/degree of belief/degree of lack-of-information/algorithms/complexity

For a long time I had a bit of difficulty understanding what "degree of belief" means. Recently I had some thoughts about it and I wonder if they make any sense, or is there some literature about ...
1
vote
2answers
102 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
1
vote
0answers
15 views

Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
1
vote
1answer
86 views

Finding the MVUE from two independent random samples

Suppose we have a random sample $X_1, X_2, \ldots, X_n$ from exponential$~(β >0)$ $\text{i.e. }f(x\mid β) = {1/β} ~e^{−x/β}$ and a random sample$~Y_1, Y_2, \ldots, Y_n$ from exponential$~(⍺ >0)...
1
vote
1answer
59 views

Multivariate conditional entropy

I would like to take data columns and compute the multivariate conditional entropy. For instance, suppose I have columns $A, B, C D, E$ and I want to compute the conditional entropy $H(E | A,B,C,D)$. ...
0
votes
0answers
16 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
60 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
1
vote
1answer
49 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
17 views

What happen to gain ratio when information gain is 0?

I am learning decision tree using C4.5, stumbled across data where its attributes has only one value, because of only one value, when calculating the information gain it resulted with 0. Because ...
2
votes
1answer
35 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
0
votes
1answer
38 views

What Is Meant by “Maximising” Posterior Probability?

My textbook says the following: The optimal coding decision (optimal in the sense of having the smallest probability of being wrong) is to find which value of $\mathbf{s}$ is most probable, ...
0
votes
0answers
33 views

Aikaike Information Criterion: derivation in original paper

I have been reading AIC paper 'Information theory and an extension of the maximum likelihood principle' by Akaike (1974). I have been able to understand up to the third section of the paper, but I am ...
1
vote
0answers
33 views

How do I measure information loss when converting categorical data to numerical?

Assume that a dataset has a mix of categorical and numerical attributes. The dataset has to undergo numeric processing which necessitates the conversion of the categorical attributes to numeric/...
3
votes
1answer
58 views

Can we apply KL divergence to the probability distributions on different domains?

When I was reading the original paper of t-SNE, I had an question whether or not we can apply KL divergence to the discrete probability distributions on different domains. In the paper, they measure ...
4
votes
2answers
305 views

What is the actual significance of a difference in AIC or BIC values?

Usually, when a difference of a statistic is discussed, that discussion is presented in the context of a significance of that difference. When self-entropy, i.e., information content, is examined, ...
0
votes
0answers
10 views

Can we calculate the best possible test set loss given the information in the training set?

Given a training set and a test set for machine learning, I am wondering if I can somehow measure if the information to predict the labels of the test set correctly is present in the training set. In ...
2
votes
1answer
64 views

reading books about Bayesian Model selection

I was trying to find some "good" reading books about Bayesian Model selection. So is there any recommendations? To be specific, I was trying to understand the Bayesian Information Criterion (BIC), the ...
3
votes
2answers
254 views
0
votes
0answers
20 views

Estimating a surprise of a word in context

What will be the best way to estimate the entropy/surprise of a word in a specific context? Let's say to compare the surprise of: context: "I watched the movie in my" word: Computer I ...
4
votes
0answers
73 views

Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...
1
vote
0answers
49 views

Why maximum likelihood estimation is same with minimizing cross-entropy? [duplicate]

Lots of articles say that MLE is as same as minimizing cross-entropy. I tried to prove this but failed. the relationship between maximizing the likelihood and minimizing the cross-entropy This ...
1
vote
0answers
20 views

How Much More Information (as bits) is Gained (if any) from a Ranking vs. Likert Scale Survey of 100 Questions's / 1-5 Scale?

All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a ...
0
votes
0answers
91 views

VC dimension of finite unions of one-sided intervals

What is the VC dimension of $k$ finite unions of one-sided intervals: If we take 3 one-sided intervals like $(-\infty, a_1] $, $(-\infty, a_2] $ and $(-\infty, a_3] $, I think union of these ...