33
votes
Statistical interpretation of Maximum Entropy Distribution
This isn't really my field, so some musings:
I will start with the concept of surprise. What does it mean to be surprised?
Usually, it means that something happened that was not expected to ...
- 66.4k
18
votes
Why is Entropy maximised when the probability distribution is uniform?
Entropy in physics and information theory are not unrelated. They're more different than the name suggests, yet there's clearly a link between. The purpose of entropy metric is to measure the amount ...
- 56.8k
16
votes
Accepted
Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian
The starred step is valid because (a) $p$ and $q$ have the same zeroth and second moments and (b) $\log(p)$ is a polynomial function of the components of $\mathbf{x}$ whose terms have total degrees $0$...
- 294k
12
votes
Is differential entropy always less than infinity?
I thought about this question some more and managed to find a counter-example, thanks also to the Piotr's comments above. The answer to the first question is no - the differential entropy of a ...
- 806
10
votes
Why is Entropy maximised when the probability distribution is uniform?
The mathematical argument is based on Jensen inequality for concave functions. That is, if $f(x)$ is a concave function on $[a,b]$ and $y_1, \ldots y_n$ are points in $[a,b]$, then:
$n \cdot f(\frac{...
- 101
8
votes
Accepted
Entropy and information content
Based on your phrasing, it seems you are equating thermodynamic entropy with information entropy. The concepts are related, but you have to be careful because they are used differently in the two ...
- 196
8
votes
What is the maximum entropy probability density function for a positive continuous variable of given mean and standard deviation?
I want to make @Glen_b's answer more explicit, here is an extra answer just because it wouldn't fit as a comment.
The formalism etc. is well explained in Chapter 11 and 12 of Jaynes' book.
Taking the ...
- 181
8
votes
Accepted
How to determine Forecastability of time series?
Parameters m and r, involved in calculation of approximate entropy (ApEn) of time series, are window (sequence) length and ...
- 8,005
8
votes
Accepted
When does the maximum likelihood correspond to a reference prior?
Correct, as long as the support of the uniform prior contains the MLE. The reason for this is that the posterior and the likelihood are proportional on the support of the uniform prior. Even if the ...
8
votes
Is there a relationship between Maximum Likelihood Estimation and the Maximum Entropy Principle?
Yes there is a connection. Namely, fitting maximum likelihood in exponential (Gibbs) family models is equivalent to doing maximum entropy with some constraints. Formally speaking, there is a primal-...
- 398
7
votes
Accepted
Intuition for the uniform distribution having the maximum entropy
Let us consider as an example a coin with $p=0.1$ provability of heads.
If we learn that the coin turned heads, our "surprise" is given by $\log_2{\frac{1}{0.1}}\approx3.3$, which is indeed greater ...
- 2,555
6
votes
Accepted
What is the maximum entropy distribution given values for several quantiles of one sample?
Maximum entropy problems do not always admit a solution. The generic expression for the maximum entropy density $f(x)$ given a set of integral constraints
\begin{equation}
\int dx \, h_i(x) \, f(x) = ...
- 96
6
votes
Accepted
What is the maximum entropy distribution given the median (instead of the mean)?
Lets look at one specific version of this problem. This is enough to show some problems with this problem formulation.
Let $X$ be a positive random variable, assumed to have an absolutely ...
- 66.4k
6
votes
Accepted
Why do we want a maximum entropy distribution, if it has the lowest information?
Because "maxent" distribution is more "in the center". A formal description of this is in this paper -- "Game Theory, Maximum Entropy, Minimum Discrepancy, and Robust Bayesian ...
- 5,403
5
votes
Accepted
Are there any contemporary uses of jackknifing?
If you take jackknifing not only to include leave-one-out but any kind of resampling-without-replacement such as $k$-fold procedures, I consider it a viable option and use it regularly, e.g. in
...
5
votes
Why is Entropy maximised when the probability distribution is uniform?
On a side note, is there any connnection between the entropy that occurs information theory and the entropy calculations in chemistry (thermodynamics) ?
Yes, there is! You can see the work of Jaynes ...
- 304
5
votes
Accepted
Support vector machines (SVMs) are the zero temperature limit of logistic regression?
In the case of hard-margin SVM and linearly separable data, this is true.
An intuitive sketch: The loss for each datapoint in logistic regression dies out almost as an exponential decay curve as you ...
- 23.3k
5
votes
Accepted
Some Questions about reference measures and maximum entropy priors (from The Bayesian Choice)
Let the author try to answer these questions about maximum entropy priors:
"If some characteristics of the prior distribution (moments,
quantiles,etc) are known, assuming that they can be written ...
- 93.5k
5
votes
Accepted
Same maximum entropy and measures
The point I am making in my book and in the previous question is not original but worth repeating. For a dominating measure $\text{d}\mu$, the maximum entropy prior is defined as maximising$$\int_\...
- 93.5k
5
votes
Proving that Shannon entropy is maximised for the uniform distribution
Using Lagrange multipliers we have the equation:
$$\mathcal{L} = \left \{ -\sum_i^k p_i \log p_i - \lambda\left ( \sum_i^k p_i - 1 \right )\right \}$$
Maximizing with respect to the probability,
$$\...
- 121
5
votes
Which has minimum concentration: the uniform distribution or the maximum entropy distribution?
However, this conflicts with the maximum entropy principle, which states that the Normal/Gaussian distribution has maximum entropy,
You're almost there, just missed one important point:
MaxEnt does ...
- 304
4
votes
Statistical interpretation of Maximum Entropy Distribution
Perhaps not exactly what you are after, but in Rissanen, J. Stochastic Complexity in Statistical Inquiry, World Scientific, 1989, p. 41 there is an interesting connection of maximum entropy, the ...
- 7,843
4
votes
Weakly informative prior distributions for scale parameters
(The question is stale, but the issue is not)
Personally, I think your intuition makes some sense. That is to say, if you don't need the mathematical tidiness of conjugacy, then whatever distribution ...
- 141
4
votes
Accepted
Relationship between entropy and information gain
// So, in view of the above I have questions which is Is mutual information another name for information gain? //
No. But MI can be expressed in terms of KL (i.e. Info Gain) http://en.wikipedia.org/...
- 903
4
votes
Accepted
Maximum likelihood estimator of joint distribution given only marginal counts
This kind of problem was studied in the paper
"Data Augmentation in Multi-way Contingency Tables With Fixed Marginal Totals" by Dobra et al (2006). Let $\theta$ denote the parameters of the model, ...
- 6,650
4
votes
Accepted
Maximum entropy distribution of a proportion with known mean and variance? Is it a beta?
It's a truncated Normal distribution. This is a consequence of Boltzmann's Theorem.
The following analysis provides the details needed to implement a practical solution.
A Normal$(\mu,\sigma)$ ...
- 294k
3
votes
Accepted
Properties of MaxEnt posterior distribution for a die with prescribed average
The MaxEnt algorithm heavily favors distributions as close to uniform as possible. Therefore, given the constraint that the average is $4$, it is more optimal to add more mass to $6$ rather than $4$ ...
3
votes
Accepted
Is it possible to use entropy maximisation to get around collinearity in linear regression?
Principal components regression is one well respected way to handle collinear predictors, and my sense is that it has a relation to entropy maximization. I don't have time right now to think this ...
- 66.5k
3
votes
Justification for invoking Maximum Entropy
First of all, I tried to comment on the question but I couldn't because I didn't have (still don't) 50 reputation, so I'm posting my opinion as an answer despite knowing it is not a complete answer to ...
- 304
3
votes
Infinite fourth moment and maximum entropy
Maximum Entropy would have me say that I should choose the Normal distribution as the one that best fit my knowledge about it.
Maximum entropy isn't really about 'best fit', it's more about about ...
- 263k
Only top scored, non community-wiki answers of a minimum length are eligible