Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [information-theory]

A branch of mathematics/statistics used to determine the information carrying capacity of a channel, whether one that is used for communication or one that is defined in an abstract sense. Entropy is one of the measures by which information theorists can quantify the uncertainty involved in predicting a random variable.

0
votes
1answer
17 views

Why dose discrete data distribution has differential entropy of negative infinity?

Recently I've been reading a paper. In section 3.1, it says "Since the discrete data distribution has differential entropy of negative infinity, this can lead to arbitrary high likelihoods even on ...
1
vote
0answers
9 views

Multi-level factor design of experiments

How should I select interactions in order to be able to encode multi-level features using interactions of main features in my design? I'm wanting to create an experimental design to be used to create ...
0
votes
0answers
11 views

Mutual Information between layer's activation and class label

I want to calculate Information gain of particular layer's activation with respect to class label, which is something quite similar to, ...
0
votes
0answers
39 views

Density Estimation Efficieny

My Question Let's say a set training samples like D from a discrete distribution like p(x) over a discrete variable vector like x is available. We don't have any prior knowledge about the form of p(x)...
0
votes
1answer
32 views

If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
0
votes
0answers
8 views

Discriminating Between Spurious and Real Relationships With Algorithmic Information

The standard means of avoiding spurious relationships is to predict them. However, it seems that this can also be accomplished with sufficiently sophisticated exploratory modeling, involving, say, ...
0
votes
0answers
17 views

Formal Definition of Over-Predictive Model

I am looking for a formal definition or criterion to determine whether or not a model is over predictive. My understanding of a model being over-predictive, is a parametric model whose parameters are ...
0
votes
0answers
14 views

Difference of mutual informations / Kullback-Leibler divergences for dependent arbitrary- and Gaussian random variables with similar second moments

Let $(Y_1, Y_2)$ be arbitrarily jointly distributed random variables, and let $(Y_{1,G}, Y_{1,G})$ be jointly distributed Gaussian random variables with the same mean and second moments as those of $(...
1
vote
0answers
42 views

Is random shuffling order preserving with respect to the entropy?

Let $X_1,X_2$ be discrete random variables such that $H(X_1)<H(X_2)$ where $H()$ is the entropy. We know that for any random mapping $T$ which is invertible and independent of a random variable $X$ ...
1
vote
0answers
46 views

How to derive a ranking function by analysing feature correlations

I am analysing some employee details to find the efficiency of the employees. Ideally I want some rankings to rank them based on these features. My features include; current salary projects ...
0
votes
1answer
8 views

The odds of matching ages being the same day of the week for 2 people (father and daughter)

What are the odds that (despite having different birthdays) my daughter (born November 16, 2006) and I (Feb 8, 1979) have had each consequestive birthday on the same day of the week, we were both born ...
0
votes
0answers
26 views

Shannon entropy as expected information content

I'm struggling with a form of viewing Shannon Entropy. Cover & Thomas say that entropy is the expected value of information content. So there is a random variable $X$ with distribution $P(X)$ and ...
0
votes
0answers
19 views

Can a Jeffreys prior be used as an Information maximizing distribution if Information is defined using differential entropy?

I know that a Jeffreys prior is the information maximizing distribution for the statistical channel. However, I want to know if I define mutual information as $$I(x;y)=h(x)-h(x|y)$$ where $h(.)$ is ...
0
votes
0answers
12 views

error in information theory

At the risk of sounding vague, someone told me that errors in information theory usually comes in the form of $c/t$, $c/t^{\alpha}$, or $\exp{(-\alpha t)}$, where $t$ is the $t-$ (or $x$ axis). I ...
1
vote
1answer
28 views

How many bits of information on average are you receiving if you get this information each time you toss a coin?

Say I can psychically predict the result of a fair coin toss with probability 0.6. On average, how many bits of accurate information am I communicating each time I do it? I think the answer is ${\...
0
votes
0answers
6 views

On the practical usage of generalisation error bounds

(This questions is based on a question that I've posted previously here, but I would like it to get more exposure) In many practical scenarios, one would like to answer how much more data is needed ...
0
votes
0answers
36 views

Connections between logistic regression, information value and Kullback-Leibler

Suppose that we are interested in modeling a binary predictor $Y=0,1$ subject to $m$ predictors $x_1,...,x_m$. First, let us examine a simpler model of the impact of $x_j$ on the response $Y$. By the ...
0
votes
0answers
24 views

Am I using conditional entropy formula correctly?

I want to use the formula for a measure of complexity of a system: $$C(X) = H(X) - \sum_{x \in X}{H(x\mid X-x)}$$ where $x$ is a subpart of the system $X,$ and $H(X)$ is shannon entropy. Then I have ...
2
votes
0answers
71 views

How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
0
votes
1answer
16 views

Analytical expression of the minimizer of cross entropy loss when the predicted function is a constant fucntion?

Let $\{y_1...y_n\} \in \{0,1\}$, and let $c \in [0,1]$. Define the cross-entropy of loss of $c$ by: $$C(c): = \sum_{j=1}^{n}- y_j ln c - (1- y_j) ln (1-c) $$. Define $c*= arg min _{c} C(c)$ Is ...
1
vote
0answers
51 views

Combining subjoint distributions to create a larger joint distribution

I am trying to construct large joint distributions through smaller joint distributions and I'm not sure how to approach the literature. I am curious if there exists a function which can take n ...
0
votes
0answers
16 views

Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
0
votes
1answer
28 views

Feature selection via conditional entropy

It looks like feature selection can be done with mutual information. Mutual information is related to conditional entropy by this equation: $I(X,Y) = H(X) - H(X|Y)$ Can we use conditional entropy ...
0
votes
0answers
19 views

Conditional entropy of an outcome

Given three discrete random variables $X$, $Y$ and $Z$, the conditional entropy $$ H(X|Y,Z) = \sum_{X}\sum_{Y}\sum_{Z} p(x,y,z) \ \text{log}\frac{p(y,z)}{p(x,y,z)} $$ If I want to calculate the ...
1
vote
2answers
96 views

Convexity of cross entropy

I am not sure if this is a better fit for this site or mathematics.stackexchange but I've seen similar questions on here before. I'd like to know if the following is true and if so, how I could go ...
0
votes
0answers
51 views

Can information entropy be computed on arbitrary set of non-negative numbers smaller than 1?

I have a set of non-negative numbers smaller than 1 which do not sum to 1. These numbers will be used in some weighted sum. I'm wondering in this case can we still measure the degree of concentration ...
0
votes
0answers
21 views

Why H(y) is a concave function of p(x) when p(y) is a linear function of p(x)?

I got stuck in the line when reading a theorem in The Elements of Information Theory on page 59: If p(y|x) is fixed, then p(y) is a linear function of p(x). Hence H(Y), which is a concave ...
1
vote
1answer
34 views

How to check if covariates in multiple regression is explaining the same?

I am a master's student doing my thesis at the moment and have come to the point of determining my empirical setup. I would like to get some guidance, in terms of what I am thinking is proper.. I ...
2
votes
1answer
126 views

Interpretation of spectral entropy of a timeseries

The tsfeatures package for R has an entropy() function. The vignette for the package describes it as: The spectral entropy is ...
3
votes
1answer
36 views

Measure of randomness that maps well to “hard to guess”?

It was recently pointed out to me that entropy does not necessarily map well to "hard to guess". E.g.: consider a distribution where there's a very high (say, 90%) chance you select a single, ...
4
votes
0answers
34 views

Uniform distribution on the simplex. - Thomas cover

I'm trying to formulate the solution for the following problem: I was thinking in finding the equivalent distribution on $X_i$ based on $Y_i$, but I think I'm cheating. I think that the autor wants ...
0
votes
0answers
21 views

Keeping starting variables after a Principal component analysis

I know the PCA theory and how keeping the k-first principal components help dimensionality reduction. But what about keeping the base vectors ? You start with $n$ row vectors $x_1,...,x_n$, ...
3
votes
0answers
34 views

Which properties yield the exponential family of distributions?

It seems like every resource that discusses exponential families simply defines the family of distributions, explains why it's useful and then derives some of its properties. I have only seen one ...
0
votes
0answers
29 views

joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
0
votes
0answers
27 views

What does Joint entropy really tells us?

I have two sequences of data A and B. A = [sunny, sunny, cloudy, rainy, sunny, sunny] with entropy 1.25 bits b = [hot, hot, cold, cold, cold, hot] with entropy 1 bits I have calculated joint ...
0
votes
0answers
21 views

Multivariate Conditional Entropy as a test of correlation between random variables

I use the word columns to mean the data from which a random variable can be estimated. It is a sample of a random variable. I am working with $N$ columns of weakly correlated data. Furthermore, I ...
2
votes
2answers
50 views

Estimating Differential entropy from unbiased samples of a probability distribution

If I can get N unbiased samples $x$ from $p(x)$ how can I approximate the Differential entropy: $$H(X) = -\displaystyle\int_{x} p(x)\log p(x) dx$$ I'm not very knowledgeable in statistics so I'm not ...
0
votes
0answers
47 views

SHAP values vs Information Gain?

SHAP values which are essentially the variable importance at a local level where each variable's importance is assessed by different in probability outcome of a model with and without the variable. I ...
0
votes
0answers
14 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
0
votes
0answers
24 views

Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
0
votes
0answers
20 views

Information gain or regression for measuring to what extent a continuous quantity detemines a discrete one?

I have a small population S and two quantities P and Q associated to its members. P is integer valued (say, age taking integer values between 1 and 20) and Q real valued (say, height in meters). I can ...
0
votes
0answers
10 views

How to assess the informativeness of sets of past observations?

I know how to forecast one timeseries given its own past values and the values of other timeseries. However, I'm currently having a problem where the past values of a timeseries are partitioned into ...
0
votes
1answer
21 views

Prior/degree of belief/degree of lack-of-information/algorithms/complexity

For a long time I had a bit of difficulty understanding what "degree of belief" means. Recently I had some thoughts about it and I wonder if they make any sense, or is there some literature about ...
1
vote
2answers
117 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
1
vote
1answer
20 views

Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
1
vote
1answer
154 views

Finding the MVUE from two independent random samples

Suppose we have a random sample $X_1, X_2, \ldots, X_n$ from exponential$~(β >0)$ $\text{i.e. }f(x\mid β) = {1/β} ~e^{−x/β}$ and a random sample$~Y_1, Y_2, \ldots, Y_n$ from exponential$~(⍺ >0)...
1
vote
1answer
98 views

Multivariate conditional entropy

I would like to take data columns and compute the multivariate conditional entropy. For instance, suppose I have columns $A, B, C D, E$ and I want to compute the conditional entropy $H(E | A,B,C,D)$. ...
0
votes
0answers
18 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
114 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...