33 votes

Statistical interpretation of Maximum Entropy Distribution

This isn't really my field, so some musings: I will start with the concept of surprise. What does it mean to be surprised? Usually, it means that something happened that was not expected to ...
user avatar
18 votes

Why is Entropy maximised when the probability distribution is uniform?

Entropy in physics and information theory are not unrelated. They're more different than the name suggests, yet there's clearly a link between. The purpose of entropy metric is to measure the amount ...
user avatar
  • 56.8k
16 votes
Accepted

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

The starred step is valid because (a) $p$ and $q$ have the same zeroth and second moments and (b) $\log(p)$ is a polynomial function of the components of $\mathbf{x}$ whose terms have total degrees $0$...
user avatar
  • 294k
12 votes

Is differential entropy always less than infinity?

I thought about this question some more and managed to find a counter-example, thanks also to the Piotr's comments above. The answer to the first question is no - the differential entropy of a ...
user avatar
  • 806
10 votes

Why is Entropy maximised when the probability distribution is uniform?

The mathematical argument is based on Jensen inequality for concave functions. That is, if $f(x)$ is a concave function on $[a,b]$ and $y_1, \ldots y_n$ are points in $[a,b]$, then: $n \cdot f(\frac{...
user avatar
8 votes
Accepted

Entropy and information content

Based on your phrasing, it seems you are equating thermodynamic entropy with information entropy. The concepts are related, but you have to be careful because they are used differently in the two ...
user avatar
8 votes

What is the maximum entropy probability density function for a positive continuous variable of given mean and standard deviation?

I want to make @Glen_b's answer more explicit, here is an extra answer just because it wouldn't fit as a comment. The formalism etc. is well explained in Chapter 11 and 12 of Jaynes' book. Taking the ...
user avatar
8 votes
Accepted

How to determine Forecastability of time series?

Parameters m and r, involved in calculation of approximate entropy (ApEn) of time series, are window (sequence) length and ...
user avatar
8 votes
Accepted

When does the maximum likelihood correspond to a reference prior?

Correct, as long as the support of the uniform prior contains the MLE. The reason for this is that the posterior and the likelihood are proportional on the support of the uniform prior. Even if the ...
user avatar
8 votes

Is there a relationship between Maximum Likelihood Estimation and the Maximum Entropy Principle?

Yes there is a connection. Namely, fitting maximum likelihood in exponential (Gibbs) family models is equivalent to doing maximum entropy with some constraints. Formally speaking, there is a primal-...
user avatar
7 votes
Accepted

Intuition for the uniform distribution having the maximum entropy

Let us consider as an example a coin with $p=0.1$ provability of heads. If we learn that the coin turned heads, our "surprise" is given by $\log_2{\frac{1}{0.1}}\approx3.3$, which is indeed greater ...
user avatar
  • 2,555
6 votes
Accepted

What is the maximum entropy distribution given values for several quantiles of one sample?

Maximum entropy problems do not always admit a solution. The generic expression for the maximum entropy density $f(x)$ given a set of integral constraints \begin{equation} \int dx \, h_i(x) \, f(x) = ...
user avatar
  • 96
6 votes
Accepted

What is the maximum entropy distribution given the median (instead of the mean)?

Lets look at one specific version of this problem. This is enough to show some problems with this problem formulation. Let $X$ be a positive random variable, assumed to have an absolutely ...
user avatar
6 votes
Accepted

Why do we want a maximum entropy distribution, if it has the lowest information?

Because "maxent" distribution is more "in the center". A formal description of this is in this paper -- "Game Theory, Maximum Entropy, Minimum Discrepancy, and Robust Bayesian ...
user avatar
5 votes
Accepted

Are there any contemporary uses of jackknifing?

If you take jackknifing not only to include leave-one-out but any kind of resampling-without-replacement such as $k$-fold procedures, I consider it a viable option and use it regularly, e.g. in ...
user avatar
5 votes

Why is Entropy maximised when the probability distribution is uniform?

On a side note, is there any connnection between the entropy that occurs information theory and the entropy calculations in chemistry (thermodynamics) ? Yes, there is! You can see the work of Jaynes ...
user avatar
5 votes
Accepted

Support vector machines (SVMs) are the zero temperature limit of logistic regression?

In the case of hard-margin SVM and linearly separable data, this is true. An intuitive sketch: The loss for each datapoint in logistic regression dies out almost as an exponential decay curve as you ...
user avatar
  • 23.3k
5 votes
Accepted

Some Questions about reference measures and maximum entropy priors (from The Bayesian Choice)

Let the author try to answer these questions about maximum entropy priors: "If some characteristics of the prior distribution (moments, quantiles,etc) are known, assuming that they can be written ...
user avatar
  • 93.5k
5 votes
Accepted

Same maximum entropy and measures

The point I am making in my book and in the previous question is not original but worth repeating. For a dominating measure $\text{d}\mu$, the maximum entropy prior is defined as maximising$$\int_\...
user avatar
  • 93.5k
5 votes

Proving that Shannon entropy is maximised for the uniform distribution

Using Lagrange multipliers we have the equation: $$\mathcal{L} = \left \{ -\sum_i^k p_i \log p_i - \lambda\left ( \sum_i^k p_i - 1 \right )\right \}$$ Maximizing with respect to the probability, $$\...
user avatar
  • 121
5 votes

Which has minimum concentration: the uniform distribution or the maximum entropy distribution?

However, this conflicts with the maximum entropy principle, which states that the Normal/Gaussian distribution has maximum entropy, You're almost there, just missed one important point: MaxEnt does ...
user avatar
4 votes

Statistical interpretation of Maximum Entropy Distribution

Perhaps not exactly what you are after, but in Rissanen, J. Stochastic Complexity in Statistical Inquiry, World Scientific, 1989, p. 41 there is an interesting connection of maximum entropy, the ...
user avatar
  • 7,843
4 votes

Weakly informative prior distributions for scale parameters

(The question is stale, but the issue is not) Personally, I think your intuition makes some sense. That is to say, if you don't need the mathematical tidiness of conjugacy, then whatever distribution ...
user avatar
4 votes
Accepted

Relationship between entropy and information gain

// So, in view of the above I have questions which is Is mutual information another name for information gain? // No. But MI can be expressed in terms of KL (i.e. Info Gain) http://en.wikipedia.org/...
user avatar
4 votes
Accepted

Maximum likelihood estimator of joint distribution given only marginal counts

This kind of problem was studied in the paper "Data Augmentation in Multi-way Contingency Tables With Fixed Marginal Totals" by Dobra et al (2006). Let $\theta$ denote the parameters of the model, ...
user avatar
  • 6,650
4 votes
Accepted

Maximum entropy distribution of a proportion with known mean and variance? Is it a beta?

It's a truncated Normal distribution. This is a consequence of Boltzmann's Theorem. The following analysis provides the details needed to implement a practical solution. A Normal$(\mu,\sigma)$ ...
user avatar
  • 294k
3 votes
Accepted

Properties of MaxEnt posterior distribution for a die with prescribed average

The MaxEnt algorithm heavily favors distributions as close to uniform as possible. Therefore, given the constraint that the average is $4$, it is more optimal to add more mass to $6$ rather than $4$ ...
user avatar
3 votes
Accepted

Is it possible to use entropy maximisation to get around collinearity in linear regression?

Principal components regression is one well respected way to handle collinear predictors, and my sense is that it has a relation to entropy maximization. I don't have time right now to think this ...
user avatar
  • 66.5k
3 votes

Justification for invoking Maximum Entropy

First of all, I tried to comment on the question but I couldn't because I didn't have (still don't) 50 reputation, so I'm posting my opinion as an answer despite knowing it is not a complete answer to ...
user avatar
3 votes

Infinite fourth moment and maximum entropy

Maximum Entropy would have me say that I should choose the Normal distribution as the one that best fit my knowledge about it. Maximum entropy isn't really about 'best fit', it's more about about ...
user avatar
  • 263k

Only top scored, non community-wiki answers of a minimum length are eligible