Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

Filter by
Sorted by
Tagged with
1 vote
1 answer
43 views

How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
user avatar
  • 11
0 votes
0 answers
23 views

Maximum entropy prior for binomial trial, is it 1/(2n+2) and this reasonable?

I am looking into what prior probability should be assigned to an event in a binomial trial that could occur but has not yet occurred after many trials. rephrased, what probability should be assigned ...
user avatar
2 votes
1 answer
200 views

Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm

I am quite familiar with Soft-Actor-Critic (SAC) and its many applications in continuous control RL environments. However, when implementing this algorithm in a practical setting, one thing that still ...
user avatar
  • 163
0 votes
0 answers
18 views

Discrete Bayes Net learning under parameter constraints

What is some relevant research available on estimating the parameters of a Bayes Net (with known structure) when there are known constraints on conditional and marginal probabilities? For example, ...
user avatar
  • 1,428
0 votes
0 answers
21 views

Differentiating entropy in Reinforcement Learning as Probabilistic Inference

I am studying the paper Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (https://arxiv.org/abs/1805.00909) and I do not understand how the author differentiate the ...
user avatar
3 votes
1 answer
86 views

Highest Entropy Distribution, on $[0,\infty)$ given Mean, Variance, and goes to $p(0) = 0$?

I am dealing with temperature measurements, and normally we assume the probability of getting a measurement $t_i$ with a certain uncertainty $\sigma_t$ given the model ('true' value) $M(x_i)$ (where $...
user avatar
  • 143
1 vote
0 answers
32 views

Maximum entropy distribution of a positive continuous variable with known mean and vanishing probability at 0

I am working on a problem where I know that the variable of concern $x$ is positive, and has no upper bound on its value and whose probability would vanish as we approach 0, $\lim_{x \rightarrow 0^+} ...
user avatar
3 votes
1 answer
30 views

How to statistically detect a treshold effect over a dependent variable measured repeated times on the same population

I want to identify the level of a predictive variable X (with Gaussian distribution) able to induce a reduction in a variable y (with Poisson distribution), that has been measured over the same ...
user avatar
0 votes
0 answers
208 views

Computing the gradient of the log-partition function in a linear-chain conditional random field (CRF) model

Query. When computing the gradient of the log-partition function for an exponential family distribution specified by the linear-chain conditional random field (CRF) model, will unary conditional ...
user avatar
  • 2,206
3 votes
0 answers
265 views

Geometric distribution and entropy

According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{ \mu} $ ...
user avatar
  • 426
1 vote
0 answers
56 views

What is the maximum entropy joint Bernoulli distribution with fixed covariances and individual means?

We have Bernoulli variables $B_i$ with known means $E(B_i)$ and covariance matrix $\Sigma = (cov(B_i, B_j))$. What joint distribution would have the maximum entropy?
user avatar
  • 111
1 vote
0 answers
55 views

How to evaluate the likelihood of a conditional MAXENT estimation?

Suppose I have a random variable $Y$ (the outcome) and a set of random variables $\mathbf{X}$ (the input variables). I don't have access to observations of the joint distribution of $P(Y, \mathbf{X})$,...
user avatar
  • 260
3 votes
0 answers
70 views

Generalization of Burg's Maximum Entropy Theorem

Burg's Theorem characterizes the form of an entropy-maximizing time series, subject to constraints on the autocorrelation. More precisely, the theorem states that the autoregressive Gaussian process $...
user avatar
  • 1,124
1 vote
0 answers
10 views

Complexity of Maximum Entropy Algorithm in Sentiment Analysis

Does anyone know how the process of calculating the complexity of the maximum entropy algorithm and its implementation later in the sentiment analysis? Please help me, because I haven't got a ...
user avatar
3 votes
0 answers
98 views

What determines the functional form of maximum entropy constraints?

I'm familiar with the maximum entropy (ME) principle in statistical mechanics, where, for example, the Boltzmann distribution $p(\epsilon_i|\beta)$ is identified as the ME distribution constrained by ...
user avatar
  • 155
1 vote
1 answer
547 views

Is there a relationship between Maximum Likelihood Estimation and the Maximum Entropy Principle?

I know that both techniques can be used to estimate distribution from the data, but I didn't see anything in common between the two and I haven't found anything yet for the internet that relates the ...
user avatar
2 votes
0 answers
72 views

Maximum entropy discrete distributions with specified mean

Consider a discrete distribution on {1, ...,n}, with mean given as $m$, what is the maximum entropy distribution? I know it takes the form $p_{X}(k)=ar^{k}$ and is a geometric distribution when n is ...
user avatar
2 votes
0 answers
41 views

entropy regularization in generative model

I am wondering if it is possible to use entropy as a regularization in a generative model. For example, in the conjugate model where $x_i \in X$ is observed data and generated from a Normal ...
user avatar
  • 707
1 vote
0 answers
63 views

If a zero entropy distribution implies high information a priori, what does it mean ex posteriori?

The following counteracts the statements made for the maximum entropy principle case in order to posit a pseudo "minimum entropy principle" case that is simply the polar opposite of the ...
user avatar
  • 3,089
0 votes
1 answer
277 views

Which has minimum concentration: the uniform distribution or the maximum entropy distribution?

For a continuous random variable, the uniform distribution has high entropy because it demonstrates the greatest level uncertainty. However, this conflicts with the maximum entropy principle, which ...
user avatar
  • 3,089
1 vote
0 answers
132 views

Multiplying vector by the covariance matrix only known approximately

(cross-posted on math.SE) For random variable $(x,y)$ in $\mathbb{R}^{2d}$ and vector $v$, I need to perform the following operation on a $d \times d$ covariance matrix $E[xy']$ $$T(v)=E[xy']v$$ The ...
user avatar
3 votes
2 answers
258 views

How does a distribution's differential entropy correspond to its moments?

The Gaussian distribution maximizes entropy for the following functional constraints $$E(x) = \mu$$ and $$E((x-\mu)^2) = \sigma^2$$ which are just its first and second statistical moments (true ...
user avatar
  • 3,089
2 votes
1 answer
651 views

Does minimizing KL-divergence result in maximum entropy principle?

The Kullback-Leibler divergence (or relative entropy) is a measure of how a probability distribution differs from another reference probability distribution. I want to know what connection it has to ...
user avatar
  • 3,089
2 votes
1 answer
731 views

What does maximizing mutual information do?

In information theory, there is something called the maximum entropy principle. Are other information measures, such as mutual information, also commonly maximized? If mutual information describes the ...
user avatar
  • 3,089
0 votes
0 answers
43 views

Is the maximum entropy probability distribution only determined through comparison?

The maximum entropy probability distribution has entropy at least as great as that of all other members of a specified class of probability distributions (pdf's). Does that mean that the pdf with ...
user avatar
  • 3,089
1 vote
2 answers
386 views

Do all random variables' probability distributions have entropy?

Entropy of probability distributions is the weighted average of the log probabilities of each observation of a random variable. Does this mean that every random variable that has a probability ...
user avatar
  • 3,089
3 votes
1 answer
236 views

Why do we want a maximum entropy distribution, if it has the lowest information?

It is said that the distribution with the largest entropy should be chosen as the least-informative default. That is, we should choose the distribution that maximizes entropy because it has the lowest ...
user avatar
  • 3,089
1 vote
0 answers
21 views

Am I understanding correctly the Maximum Entropy concept using a sentence?

In the sentence "The house is white", each word carries a different amount of information. If I remove the "The" from the sentence, almost nothing happens: you are still able to ...
user avatar
1 vote
1 answer
78 views

Continuous Entropy and Maximum Entropy Solution

This is a problem that I have been working on and the mathematics of it have me fairly stumped. I am given the continuous entropy for a density $p(x)$. It is $H(X)=-\int_{0}^{\infty}p(x)\text{log}\: ...
user avatar
2 votes
1 answer
38 views

Is this implementation detail for solving maximum entropy on a computer correct?

I am currently looking at a paper by Mattos and Veiga, who describe an approach to solving the maximum entropy problem subject to linear constraints: $$\begin{aligned} \max_{p_i} -\sum_{i=1}^N p_i \...
user avatar
  • 1,501
1 vote
1 answer
246 views

When is uniform distribution have maximum entropy instead of normal distribution?

As far as I know, when we have just data and no constraints (other than probabilities must add up to 1), the distribution that gives maximum entropy is uniform distribution. But when we know mean and ...
user avatar
4 votes
0 answers
92 views

Why are $\mathbb{E}( \ln(x))$ and $\mathbb{E} ( \ln(1 - x))$ reasonable descriptions of knowledge about a beta distribution?

The max entropy philosophy states that given some constraints on the prior, we should choose the prior that is maximum entropy subject to those constraints. I know that the Beta($\alpha, \beta$) is ...
user avatar
  • 211
7 votes
0 answers
99 views

Do Lévy α-stable distributions maximize entropy subject to a simple constraint?

Is there a simple constraint on real-valued distributions such that the maximum entropy distribution is Lévy α-stable? Special cases include the Normal and Cauchy distributions for which the answer is ...
user avatar
  • 211
1 vote
2 answers
137 views

How do I prove conditional entropy is a good measure of information?

This question is a follow-up of Does "expected entropy" make sense?, which you don't have to read as I'll reproduce the relevant parts. Let's begin with the statement of the problem A student has ...
user avatar
  • 1,427
3 votes
0 answers
206 views

Are there nonparametric generative models for datasets?

Typically when I see generative models, e.g., Latent Dirichlet Allocation (JMLR) or Linear/Quadratic Discriminant Analysis (wikipedia LDA), they are probabilistic models that belong to the exponential ...
user avatar
4 votes
1 answer
465 views

Intuition for the uniform distribution having the maximum entropy

I saw the following explanation for Entropy in probability: (Entropy). The surprise of learning that an event with probability $p$ happened is defined as $\log_2(1/p)$, measured in a unit called ...
user avatar
1 vote
1 answer
465 views

MaxEnt model vs cross entropy loss

Pardon my ignorance. I am still learning. We try to minimize the cross-entropy loss for best results. However, why should the entropy be high for a MaxEnt model for the model to be good? My ...
user avatar
  • 11
0 votes
0 answers
29 views

Maximum entropy function, with f(0)=0?

I want to derive the Maximum Entropy distribution (f(x)) with the following constraints: 1. non-negative 2. specified mean 3. specified variance 4. f(0)=0 I know how to derive the MaxEnt distro with ...
user avatar
1 vote
0 answers
17 views

Maximum entropy prior for dichotomous variables [closed]

I have a set of dichotomous variables $A, B, C,$... and I know their probabilities $P(A), P(B), P(C),$... as well es their pairwise dependencies $P(A \cap B), P(A \cap C), P(B \cap C),$... . Or in ...
user avatar
1 vote
0 answers
30 views

Why generative models in Machine Learning are Boltzmann distribution-backed?

I learned from this review paper that MaxEnt models naturally display a Boltzmann distribution for the data samples, it comes from the Principle of Maximum Entropy. But I could not understand why this ...
user avatar
  • 11
6 votes
0 answers
101 views

How can a probability densitiy be estimated based on the maximum entropy principle, with constraints in the order statistics?

Let's say we are given a sample $\{ z_i \}$, $i=1,2,\ldots,N$, such that each value $z_i$ corresponds to the minimum of $n$ random variables $x$, i.e., $z = \min \{ x_1, x_2,\ldots,x_n \}$. The ...
user avatar
  • 1,568
0 votes
0 answers
70 views

How to minimise and expectation with respect to a parameter?

Suppose that $X$ is a random variable with distribution $G$. Let $H(X;\theta)$ be a parametric function with $\theta \in \Theta \subset {\mathbb R}^p$. I want to maximize the function $$\varphi(\...
user avatar
  • 1
2 votes
0 answers
21 views

Maximum entropy prior for r.v. supported on real line with no other constraints?

What would be a suitable maximum entropy prior for a random variable supported on the real line with no other constraints (i.e. unknown mean, unknown variance, unknown bounds)? All kinds of answers (...
user avatar
1 vote
1 answer
152 views

how to calculate entropy on matrix of words, topics

I have been digging in the concept of entropy for a while, now it comes to the implementation part I feel I am confused. Imagine that we have a matrix 20 * 3 standing for 20 words 3 topics (by 20 ...
user avatar
  • 228
2 votes
0 answers
56 views

Determining a probability distribution from constraints on where its mass is

Let $X$ be a random variable over the real line. Suppose that we know that $X$ is a Pearson distribution. Furthermore, suppose we know how the mass of $X$ is distributed into 6 intervals, so that if $...
user avatar
3 votes
0 answers
83 views

Maximum entropy probability distribution over non-negative support and finite mean?

I'm trying to derive which univariate probability distribution maximizes entropy, assuming finite mean $\mu$ and non-negative support $[0, \infty)$. I know that the answer is the exponential ...
user avatar
3 votes
2 answers
2k views

Proving that Shannon entropy is maximised for the uniform distribution

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further ...
user avatar
  • 39
1 vote
1 answer
468 views

Estimating maximum entropy distribution given first n moments

Is there a good way to estimate the pdf, pdf up to a constant multiple, cdf, or quantile function of a distribution given the first n moments? A closed form for one of those functions in terms of the ...
user avatar
2 votes
1 answer
140 views

Is the marginal distribution of a maximum entropy distribution also a maximum entropy distribution?

I am considering under what circumstances maximum entropy distributions are closed under marginalization. The main case I am interested is the following setting: a finite set of discrete random ...
user avatar
6 votes
1 answer
115 views

Maximum entropy distribution on the hypercube

Given the first two moments, the maximum entropy distribution over $\mathbb{R}$ is known the be the normal distribution. What is the analogue for a distribution over $[0,1]$ given either only the ...
user avatar
  • 527