# Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

168 questions
Filter by
Sorted by
Tagged with
31 views

### Under What Conditions Does a Gaussian Mixture Model (GMM) Have Maximum Entropy?

Introduction I'm delving into Gaussian Mixture Models (GMMs) within unsupervised learning frameworks and am particularly interested in their statistical properties, with a focus on entropy. Entropy ...
1 vote
24 views

### Exponential families as families of limite distributions of Markov processes

An exponential family verifies a maximum entropy property: each density is the maximum entropy density given the expectation of its sufficient statistic. On the other hand, from my understanding, the ...
• 151
11 views

### What is the relationship between maximum entropy sampling and bayesian d-optimal design?

In regards to optimal design, specifically sequential design or active learning for parameter inference (e.g. linear regression). As I understand it, Bayesian D-optimal design maximises the expected ...
27 views

### Maximum Entropy distribution of a ticking clock

Say I have a clock that emits "ticks". An ideal clock looks like a dirac comb. It has: perfect periodicity of ticks (there is a precise fixed time interval between any two consecutive ticks)...
• 255
57 views

### Minimizing cross entropy over a restricted domain?

Suppose $f(x;q)$ is the true distribution. The support of the random variable $X$ is $\Omega$. Suppose, I am interested in a particular subset of $\Xi \subset \Omega$. I would like to minimize the ...
• 19
185 views

### When and how was the Bernoulli distribution with real binomial proportion introduced?

I certainly should read Jakob Bernoulli's Ars Conjectandi again but let me share my concerns. I'm just wondering when and how the Bernoulli distribution $Be(p)$ (and related distributions like the ...
• 39
93 views

### Does every distribution family have a set of maximum entropy constraints?

I am reflecting on these examples of maximum entropy distributions. I am (pleasantly) surprised that various common distribution families have maximum entropy constraints. It got me wondering if ...
• 9,361
1 vote
96 views

### Is the principle of maximum entropy misleading?

If a distribution belongs to a certain class, then the distribution with the largest entropy in that class is typically referred to as the least-informative distribution. To me, this his highly ...
191 views

### What is the reasoning behind max entropy constraints for the gamma distribution?

The max entropy method is a way of deriving a probability distribution given only the information you know about how the data is distributed and nothing more. For example the normal distribution can ...
• 171
552 views

### Jaynes' Description of Maximum Entropy Distribution

I am reading E. T. Jaynes' probability theory book, and I am at chapter 11 where he introduces the maximum entropy principle. I understand that Jaynes separates the notion of probability from that of ...
• 197
1 vote
140 views

### How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
• 11
2k views

### Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm

I am quite familiar with Soft-Actor-Critic (SAC) and its many applications in continuous control RL environments. However, when implementing this algorithm in a practical setting, one thing that still ...
• 205
1 vote
84 views

### Discrete Bayes Net learning under parameter constraints

What is some relevant research available on estimating the parameters of a Bayes Net (with known structure) when there are known constraints on conditional and marginal probabilities? For example, ...
• 1,158
196 views

34 views

### How to statistically detect a treshold effect over a dependent variable measured repeated times on the same population

I want to identify the level of a predictive variable X (with Gaussian distribution) able to induce a reduction in a variable y (with Poisson distribution), that has been measured over the same ...
• 570
395 views

### Computing the gradient of the log-partition function in a linear-chain conditional random field (CRF) model

Query. When computing the gradient of the log-partition function for an exponential family distribution specified by the linear-chain conditional random field (CRF) model, will unary conditional ...
• 2,550
419 views

### Geometric distribution and entropy

According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{ \mu}$ ...
• 676
1 vote
112 views

### What is the maximum entropy joint Bernoulli distribution with fixed covariances and individual means?

We have Bernoulli variables $B_i$ with known means $E(B_i)$ and covariance matrix $\Sigma = (cov(B_i, B_j))$. What joint distribution would have the maximum entropy?
• 111
1 vote
83 views

### How to evaluate the likelihood of a conditional MAXENT estimation?

Suppose I have a random variable $Y$ (the outcome) and a set of random variables $\mathbf{X}$ (the input variables). I don't have access to observations of the joint distribution of $P(Y, \mathbf{X})$,...
• 336
113 views

• 83