# Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

71 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
100 views

### Do Lévy α-stable distributions maximize entropy subject to a simple constraint?

Is there a simple constraint on real-valued distributions such that the maximum entropy distribution is Lévy α-stable? Special cases include the Normal and Cauchy distributions for which the answer is ...
1k views

### Logistic regression and maximum entropy

I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier. My question is, how does one end up with the formula for logistic regression starting with ...
2k views

### Maximum entropy classifier and sentiment analysis

I am doing a project work in sentiment analysis (on Twitter data) using machine learning approach. In order to find the 'best' way to this I have experimented with naive Bayesian and maximum entropy ...
104 views

### How can a probability densitiy be estimated based on the maximum entropy principle, with constraints in the order statistics?

Let's say we are given a sample $\{ z_i \}$, $i=1,2,\ldots,N$, such that each value $z_i$ corresponds to the minimum of $n$ random variables $x$, i.e., $z = \min \{ x_1, x_2,\ldots,x_n \}$. The ...
269 views

### MCMC for Maximum Entropy?

Is there a way to sample from a discrete probability distribution, whose distribution itself is the solution to a Maximum Entropy problem with known linear constraints, without needing to solve for ...
477 views

### Reconstructing joint distribution from marginals

I think this is a rather open question. Suppose I have bi-dimensional data $(x_i, y_i)$. I have some reasonable model for the marginals, say distributions $F_X$ and $F_Y$ (parametric). How to ...
94 views

### Why are $\mathbb{E}( \ln(x))$ and $\mathbb{E} ( \ln(1 - x))$ reasonable descriptions of knowledge about a beta distribution?

The max entropy philosophy states that given some constraints on the prior, we should choose the prior that is maximum entropy subject to those constraints. I know that the Beta($\alpha, \beta$) is ...
79 views

98 views

### What determines the functional form of maximum entropy constraints?

I'm familiar with the maximum entropy (ME) principle in statistical mechanics, where, for example, the Boltzmann distribution $p(\epsilon_i|\beta)$ is identified as the ME distribution constrained by ...
210 views

### Are there nonparametric generative models for datasets?

Typically when I see generative models, e.g., Latent Dirichlet Allocation (JMLR) or Linear/Quadratic Discriminant Analysis (wikipedia LDA), they are probabilistic models that belong to the exponential ...
83 views

### Maximum entropy probability distribution over non-negative support and finite mean?

I'm trying to derive which univariate probability distribution maximizes entropy, assuming finite mean $\mu$ and non-negative support $[0, \infty)$. I know that the answer is the exponential ...
178 views

86 views

### Normal max entropy

In the following link, at the bottom of the page. https://www.dsprelated.com/freebooks/sasp/Maximum_Entropy_Property_Gaussian.html The Gaussian maximum entropy is derived. I have a question about ...
525 views

### How can I maximise binary cross entropy loss?

I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
27 views

### Incorporating knowledge of aggregate outcomes to constrain predictions on finer scales

I've got county-level longitudinal data on the timing of an event between the years 1998 and 2012, and I want to use it to form a predictive model for the time that that event will occur in future ...
184 views

### Interpreting base measure in exponential family as an improper prior (because entropy)

Long-time listener, first-time caller. I'm reading the Wikipedia pages on exponential families and maximum entropy probability distributions, and trying to wrap my head round the role of the base ...
68 views

160 views

### Maximum Entropy with no index

This is a simpler problem than trying to solve, but have a feeling once get the methodology I can apply it to the harder problem. Let $H(p)= -q \ln(q) - p \ln(p)$ be the entropy of the Bernoulli ...
1 vote
49 views

### How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
1 vote
37 views

1 vote
489 views

### MaxEnt model vs cross entropy loss

Pardon my ignorance. I am still learning. We try to minimize the cross-entropy loss for best results. However, why should the entropy be high for a MaxEnt model for the model to be good? My ...
1 vote
30 views

### Why generative models in Machine Learning are Boltzmann distribution-backed?

I learned from this review paper that MaxEnt models naturally display a Boltzmann distribution for the data samples, it comes from the Principle of Maximum Entropy. But I could not understand why this ...
1 vote
164 views

### how to calculate entropy on matrix of words, topics

I have been digging in the concept of entropy for a while, now it comes to the implementation part I feel I am confused. Imagine that we have a matrix 20 * 3 standing for 20 words 3 topics (by 20 ...
1 vote
45 views

### Where is my misunderstanding/mistake in calculation about maximum entropy

I am looking through the book Philosophy of Science : Entropy and Uncertainty by Teddy Seidenfeld and have some questions. He talks about the basic dice example, where by you have a fair six sided ...
1 vote
783 views

### Maximum Entropy Bootstrap - how to implement?

I have a question about the Maximum Entropy Bootstrap algorithm for time series. I'm confused about how to implement step 5, where you compute the sample quantiles of the Maximum Entropy ...
1 vote
I'm trying to construct a prior probability density function, $f_X(x)$, for a fat-tailed distribution using the maximum entropy (MaxEnt) method. For my known "testable" information, I have the ...