# Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

168 questions
Filter by
Sorted by
Tagged with
47 views

### Minimizing cross entropy over a restricted domain?

Suppose $f(x;q)$ is the true distribution. The support of the random variable $X$ is $\Omega$. Suppose, I am interested in a particular subset of $\Xi \subset \Omega$. I would like to minimize the ...
158 views

### When and how was the Bernoulli distribution with real binomial proportion introduced?

I certainly should read Jakob Bernoulli's Ars Conjectandi again but let me share my concerns. I'm just wondering when and how the Bernoulli distribution $Be(p)$ (and related distributions like the ...
55 views

### Does every distribution family have a set of maximum entropy constraints?

I am reflecting on these examples of maximum entropy distributions. I am (pleasantly) surprised that various common distribution families have maximum entropy constraints. It got me wondering if ...
33 views

### Are there maximum entropy distributions with fixed moments of a certain order?

A characterization of the multivariate Gaussian distribution with a fixed mean and covariance matrix is that it is the unique probability distribution that maximizes differential entropy. That is, the ...
1 vote
71 views

### Is the principle of maximum entropy misleading?

If a distribution belongs to a certain class, then the distribution with the largest entropy in that class is typically referred to as the least-informative distribution. To me, this his highly ...
35 views

### Why do we want to constrain E[ln(x)] in some maximum entropy models?

If we look at the table of distributions in the exponential family, we will see some sufficient statistics have $\log(x)$, which means we have put constraints on $\mathbb{E}[\log(X)]$ when formulating ...
1 vote
86 views

### What is the reasoning behind max entropy constraints for the gamma distribution?

The max entropy method is a way of deriving a probability distribution given only the information you know about how the data is distributed and nothing more. For example the normal distribution can ...
459 views

### Jaynes' Description of Maximum Entropy Distribution

So I am reading E. T. Jaynes probability theory book, and I am at chapter 11 where he introduces the maximum entropy principle. I understand that Jaynes separates the notion of probability from that ...
134 views

### Why does the target entropy value in Soft Actor Critic not use a log function?

I wanted to follow up on this prior question: Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm In particular I'm confused why that target entropy value doesn't take the log of ...
1 vote
109 views

### How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
1k views

### Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm

I am quite familiar with Soft-Actor-Critic (SAC) and its many applications in continuous control RL environments. However, when implementing this algorithm in a practical setting, one thing that still ...
1 vote
84 views

### Discrete Bayes Net learning under parameter constraints

What is some relevant research available on estimating the parameters of a Bayes Net (with known structure) when there are known constraints on conditional and marginal probabilities? For example, ...
173 views

34 views

### How to statistically detect a treshold effect over a dependent variable measured repeated times on the same population

I want to identify the level of a predictive variable X (with Gaussian distribution) able to induce a reduction in a variable y (with Poisson distribution), that has been measured over the same ...
341 views

### Computing the gradient of the log-partition function in a linear-chain conditional random field (CRF) model

Query. When computing the gradient of the log-partition function for an exponential family distribution specified by the linear-chain conditional random field (CRF) model, will unary conditional ...
329 views

### Geometric distribution and entropy

According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{ \mu}$ ...
1 vote
98 views

### What is the maximum entropy joint Bernoulli distribution with fixed covariances and individual means?

We have Bernoulli variables $B_i$ with known means $E(B_i)$ and covariance matrix $\Sigma = (cov(B_i, B_j))$. What joint distribution would have the maximum entropy?
1 vote
77 views

### How to evaluate the likelihood of a conditional MAXENT estimation?

Suppose I have a random variable $Y$ (the outcome) and a set of random variables $\mathbf{X}$ (the input variables). I don't have access to observations of the joint distribution of $P(Y, \mathbf{X})$,...
92 views