Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

Filter by
Sorted by
Tagged with
0 votes
0 answers
46 views

Discrete Bayes Net learning under parameter constraints

What is some relevant research available on estimating the parameters of a Bayes Net (with known structure) when there are known constraints on conditional and marginal probabilities? For example, ...
1 vote
1 answer
493 views

MaxEnt model vs cross entropy loss

Pardon my ignorance. I am still learning. We try to minimize the cross-entropy loss for best results. However, why should the entropy be high for a MaxEnt model for the model to be good? My ...
1 vote
1 answer
251 views

Regularization versus feature reduction in species distribution modeling using Maxent

I am wondering if there is a need to set the beta multiplier in Maxent (species distriubition modeling approach) if one is also reducing features using a contribution threshold. I have seen a number ...
20 votes
3 answers
5k views

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to ...
1 vote
1 answer
50 views

How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
2 votes
1 answer
284 views

Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm

I am quite familiar with Soft-Actor-Critic (SAC) and its many applications in continuous control RL environments. However, when implementing this algorithm in a practical setting, one thing that still ...
50 votes
8 answers
68k views

Why is Entropy maximised when the probability distribution is uniform?

I know that entropy is the measure of randomness of a process/variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i)) $. In ...
0 votes
0 answers
23 views

Maximum entropy prior for binomial trial, is it 1/(2n+2) and this reasonable?

I am looking into what prior probability should be assigned to an event in a binomial trial that could occur but has not yet occurred after many trials. rephrased, what probability should be assigned ...
0 votes
0 answers
21 views

Differentiating entropy in Reinforcement Learning as Probabilistic Inference

I am studying the paper Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (https://arxiv.org/abs/1805.00909) and I do not understand how the author differentiate the ...
3 votes
1 answer
95 views

Highest Entropy Distribution, on $[0,\infty)$ given Mean, Variance, and goes to $p(0) = 0$?

I am dealing with temperature measurements, and normally we assume the probability of getting a measurement $t_i$ with a certain uncertainty $\sigma_t$ given the model ('true' value) $M(x_i)$ (where $...
1 vote
0 answers
38 views

Maximum entropy distribution of a positive continuous variable with known mean and vanishing probability at 0

I am working on a problem where I know that the variable of concern $x$ is positive, and has no upper bound on its value and whose probability would vanish as we approach 0, $\lim_{x \rightarrow 0^+} ...
1 vote
1 answer
151 views

When is the conditional differential entropy, $h(X+Z_1\mid X+Z_2)$, maximized?

Let $Z_1$ & $Z_2$ be 2 i.i.d. RVs, each distributed according to $N(0,1)$, and let $X$ be an arbitrary RV with unit variance. What distribution of X will maximize this conditional differential ...
2 votes
1 answer
657 views

Is there a relationship between Maximum Likelihood Estimation and the Maximum Entropy Principle?

I know that both techniques can be used to estimate distribution from the data, but I didn't see anything in common between the two and I haven't found anything yet for the internet that relates the ...
3 votes
1 answer
32 views

How to statistically detect a treshold effect over a dependent variable measured repeated times on the same population

I want to identify the level of a predictive variable X (with Gaussian distribution) able to induce a reduction in a variable y (with Poisson distribution), that has been measured over the same ...
2 votes
1 answer
812 views

What does maximizing mutual information do?

In information theory, there is something called the maximum entropy principle. Are other information measures, such as mutual information, also commonly maximized? If mutual information describes the ...
2 votes
1 answer
689 views

Does minimizing KL-divergence result in maximum entropy principle?

The Kullback-Leibler divergence (or relative entropy) is a measure of how a probability distribution differs from another reference probability distribution. I want to know what connection it has to ...
0 votes
0 answers
213 views

Computing the gradient of the log-partition function in a linear-chain conditional random field (CRF) model

Query. When computing the gradient of the log-partition function for an exponential family distribution specified by the linear-chain conditional random field (CRF) model, will unary conditional ...
3 votes
0 answers
270 views

Geometric distribution and entropy

According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{ \mu} $ ...
1 vote
0 answers
58 views

How to evaluate the likelihood of a conditional MAXENT estimation?

Suppose I have a random variable $Y$ (the outcome) and a set of random variables $\mathbf{X}$ (the input variables). I don't have access to observations of the joint distribution of $P(Y, \mathbf{X})$,...
1 vote
0 answers
66 views

What is the maximum entropy joint Bernoulli distribution with fixed covariances and individual means?

We have Bernoulli variables $B_i$ with known means $E(B_i)$ and covariance matrix $\Sigma = (cov(B_i, B_j))$. What joint distribution would have the maximum entropy?
3 votes
0 answers
71 views

Generalization of Burg's Maximum Entropy Theorem

Burg's Theorem characterizes the form of an entropy-maximizing time series, subject to constraints on the autocorrelation. More precisely, the theorem states that the autoregressive Gaussian process $...
5 votes
2 answers
243 views

Is it possible to get a negative infinite differential entropy without delta function and limit?

If $f(x)=\frac{1}{x\ln{x}^2}, x\ge e$, $h(X)=+\infty$. But if I hope to let $h(X)=-\infty$, can I find such a function $f(x)$ without using limit and delta function?
1 vote
0 answers
11 views

Complexity of Maximum Entropy Algorithm in Sentiment Analysis

Does anyone know how the process of calculating the complexity of the maximum entropy algorithm and its implementation later in the sentiment analysis? Please help me, because I haven't got a ...
1 vote
1 answer
256 views

When is uniform distribution have maximum entropy instead of normal distribution?

As far as I know, when we have just data and no constraints (other than probabilities must add up to 1), the distribution that gives maximum entropy is uniform distribution. But when we know mean and ...
3 votes
1 answer
138 views

Maximum entropy distribution $> 0$ with vanishing probability at zero?

I know that the maximum entropy distribution if $x > 0$ and the mean is known is the exponential distribution. However, a large percentage of the probability for this distribution is close to zero (...
3 votes
0 answers
98 views

What determines the functional form of maximum entropy constraints?

I'm familiar with the maximum entropy (ME) principle in statistical mechanics, where, for example, the Boltzmann distribution $p(\epsilon_i|\beta)$ is identified as the ME distribution constrained by ...
7 votes
0 answers
100 views

Do Lévy α-stable distributions maximize entropy subject to a simple constraint?

Is there a simple constraint on real-valued distributions such that the maximum entropy distribution is Lévy α-stable? Special cases include the Normal and Cauchy distributions for which the answer is ...
3 votes
1 answer
446 views

What is the maximum entropy distribution given the median (instead of the mean)?

Given that the median seems to be a more robust statistic than the mean/average, I was wondering if there is a solution of the maximum entropy distribution given the median (or the median and some ...
2 votes
0 answers
75 views

Maximum entropy discrete distributions with specified mean

Consider a discrete distribution on {1, ...,n}, with mean given as $m$, what is the maximum entropy distribution? I know it takes the form $p_{X}(k)=ar^{k}$ and is a geometric distribution when n is ...
2 votes
0 answers
44 views

entropy regularization in generative model

I am wondering if it is possible to use entropy as a regularization in a generative model. For example, in the conjugate model where $x_i \in X$ is observed data and generated from a Normal ...
1 vote
0 answers
78 views

If a zero entropy distribution implies high information a priori, what does it mean ex posteriori?

The following counteracts the statements made for the maximum entropy principle case in order to posit a pseudo "minimum entropy principle" case that is simply the polar opposite of the ...
3 votes
1 answer
260 views

Why do we want a maximum entropy distribution, if it has the lowest information?

It is said that the distribution with the largest entropy should be chosen as the least-informative default. That is, we should choose the distribution that maximizes entropy because it has the lowest ...
30 votes
4 answers
4k views

Statistical interpretation of Maximum Entropy Distribution

I have used the principle of maximum entropy to justify the use of several distributions in various settings; however, I have yet to be able to formulate a statistical, as opposed to information-...
0 votes
1 answer
286 views

Which has minimum concentration: the uniform distribution or the maximum entropy distribution?

For a continuous random variable, the uniform distribution has high entropy because it demonstrates the greatest level uncertainty. However, this conflicts with the maximum entropy principle, which ...
2 votes
0 answers
140 views

Multiplying vector by the covariance matrix only known approximately

(cross-posted on math.SE) For random variable $(x,y)$ in $\mathbb{R}^{2d}$ and vector $v$, I need to perform the following operation on a $d \times d$ covariance matrix $E[xy']$ $$T(v)=E[xy']v$$ The ...
3 votes
2 answers
272 views

How does a distribution's differential entropy correspond to its moments?

The Gaussian distribution maximizes entropy for the following functional constraints $$E(x) = \mu$$ and $$E((x-\mu)^2) = \sigma^2$$ which are just its first and second statistical moments (true ...
1 vote
2 answers
426 views

Do all random variables' probability distributions have entropy?

Entropy of probability distributions is the weighted average of the log probabilities of each observation of a random variable. Does this mean that every random variable that has a probability ...
0 votes
0 answers
43 views

Is the maximum entropy probability distribution only determined through comparison?

The maximum entropy probability distribution has entropy at least as great as that of all other members of a specified class of probability distributions (pdf's). Does that mean that the pdf with ...
13 votes
1 answer
290 views

Maximum Entropy with bounded constraints

Assume we have the problem of estimating the probabilities $\{p_1,p_2,p_3\}$ subject to: $$0 \le p_1 \le .5$$ $$0.2 \le p_2 \le .6$$ $$0.3 \le p_3 \le .4$$ with only the natural constraint of $p_1+...
1 vote
0 answers
21 views

Am I understanding correctly the Maximum Entropy concept using a sentence?

In the sentence "The house is white", each word carries a different amount of information. If I remove the "The" from the sentence, almost nothing happens: you are still able to ...
1 vote
1 answer
80 views

Continuous Entropy and Maximum Entropy Solution

This is a problem that I have been working on and the mathematics of it have me fairly stumped. I am given the continuous entropy for a density $p(x)$. It is $H(X)=-\int_{0}^{\infty}p(x)\text{log}\: ...
2 votes
1 answer
38 views

Is this implementation detail for solving maximum entropy on a computer correct?

I am currently looking at a paper by Mattos and Veiga, who describe an approach to solving the maximum entropy problem subject to linear constraints: $$\begin{aligned} \max_{p_i} -\sum_{i=1}^N p_i \...
2 votes
1 answer
205 views

Understanding parameter learning use of Principle of maximum entropy in Bayesian networks

I was reading Bayesian network on wiki: https://en.wikipedia.org/wiki/Bayesian_network And It stated for parameter learning use of "Principle_of_maximum_entropy"....
6 votes
1 answer
122 views

Maximum entropy distribution on the hypercube

Given the first two moments, the maximum entropy distribution over $\mathbb{R}$ is known the be the normal distribution. What is the analogue for a distribution over $[0,1]$ given either only the ...
24 votes
4 answers
6k views

Weakly informative prior distributions for scale parameters

I have been using log normal distributions as prior distributions for scale parameters (for normal distributions, t distributions etc.) when I have a rough idea about what the scale should be, but ...
0 votes
0 answers
29 views

Maximum entropy function, with f(0)=0?

I want to derive the Maximum Entropy distribution (f(x)) with the following constraints: 1. non-negative 2. specified mean 3. specified variance 4. f(0)=0 I know how to derive the MaxEnt distro with ...
1 vote
0 answers
67 views

Understanding the composition law of maximum entropy

Shannon derived the principle of maximum entropy from 3 assumptions. One of those assumptions is the composition law. In other words, he derived that maximizing the probability weighted log of ...
5 votes
0 answers
96 views

Why are $\mathbb{E}( \ln(x))$ and $\mathbb{E} ( \ln(1 - x))$ reasonable descriptions of knowledge about a beta distribution?

The max entropy philosophy states that given some constraints on the prior, we should choose the prior that is maximum entropy subject to those constraints. I know that the Beta($\alpha, \beta$) is ...
2 votes
1 answer
102 views

"Entropy" in Fantasy Football League

Compare playoff structures of two leagues. One (League A) in which we add divisions (where the top finishers in a division move on to the playoffs based on overall-record) and the other (League B) ...
3 votes
2 answers
2k views

Proving that Shannon entropy is maximised for the uniform distribution

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further ...