Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

71 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7 votes
0 answers
100 views

Do Lévy α-stable distributions maximize entropy subject to a simple constraint?

Is there a simple constraint on real-valued distributions such that the maximum entropy distribution is Lévy α-stable? Special cases include the Normal and Cauchy distributions for which the answer is ...
user avatar
  • 211
7 votes
0 answers
1k views

Logistic regression and maximum entropy

I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier. My question is, how does one end up with the formula for logistic regression starting with ...
user avatar
  • 3,588
7 votes
0 answers
2k views

Maximum entropy classifier and sentiment analysis

I am doing a project work in sentiment analysis (on Twitter data) using machine learning approach. In order to find the 'best' way to this I have experimented with naive Bayesian and maximum entropy ...
user avatar
  • 171
6 votes
0 answers
104 views

How can a probability densitiy be estimated based on the maximum entropy principle, with constraints in the order statistics?

Let's say we are given a sample $\{ z_i \}$, $i=1,2,\ldots,N$, such that each value $z_i$ corresponds to the minimum of $n$ random variables $x$, i.e., $z = \min \{ x_1, x_2,\ldots,x_n \}$. The ...
user avatar
  • 1,578
5 votes
0 answers
269 views

MCMC for Maximum Entropy?

Is there a way to sample from a discrete probability distribution, whose distribution itself is the solution to a Maximum Entropy problem with known linear constraints, without needing to solve for ...
user avatar
  • 1,297
5 votes
0 answers
477 views

Reconstructing joint distribution from marginals

I think this is a rather open question. Suppose I have bi-dimensional data $(x_i, y_i)$. I have some reasonable model for the marginals, say distributions $F_X$ and $F_Y$ (parametric). How to ...
user avatar
  • 205
4 votes
0 answers
94 views

Why are $\mathbb{E}( \ln(x))$ and $\mathbb{E} ( \ln(1 - x))$ reasonable descriptions of knowledge about a beta distribution?

The max entropy philosophy states that given some constraints on the prior, we should choose the prior that is maximum entropy subject to those constraints. I know that the Beta($\alpha, \beta$) is ...
user avatar
  • 211
4 votes
0 answers
79 views

Distribution (CDF) estimation for strictly increasing, continuous distribution with compact support

For all $t\in 1,\dots,T$, suppose $x_t\in [0,1]$ is a draw from a distribution with unknown CDF $F:[0,1]\rightarrow [0,1]$. For future use, define $\tilde{x}\in [0,1]^T$ to be a vector containing $x_1,...
user avatar
  • 475
4 votes
0 answers
197 views

Maximum entropy priors in infinite dimensional spaces

Has the idea of a maximum entropy probability distribution been explored for function spaces, and if so what are some key papers, books, or terms to look for? For $\mathbb{R}^n$ (and discrete spaces),...
user avatar
  • 1,153
3 votes
0 answers
269 views

Geometric distribution and entropy

According to wikipedia, among all discrete probability distributions supported on $\{1, 2, 3, ... \}$ with given expected value $\mu$, the geometric distribution X with parameter $p = \frac{1}{ \mu} $ ...
user avatar
  • 376
3 votes
0 answers
71 views

Generalization of Burg's Maximum Entropy Theorem

Burg's Theorem characterizes the form of an entropy-maximizing time series, subject to constraints on the autocorrelation. More precisely, the theorem states that the autoregressive Gaussian process $...
user avatar
  • 1,154
3 votes
0 answers
98 views

What determines the functional form of maximum entropy constraints?

I'm familiar with the maximum entropy (ME) principle in statistical mechanics, where, for example, the Boltzmann distribution $p(\epsilon_i|\beta)$ is identified as the ME distribution constrained by ...
user avatar
  • 155
3 votes
0 answers
210 views

Are there nonparametric generative models for datasets?

Typically when I see generative models, e.g., Latent Dirichlet Allocation (JMLR) or Linear/Quadratic Discriminant Analysis (wikipedia LDA), they are probabilistic models that belong to the exponential ...
user avatar
3 votes
0 answers
83 views

Maximum entropy probability distribution over non-negative support and finite mean?

I'm trying to derive which univariate probability distribution maximizes entropy, assuming finite mean $\mu$ and non-negative support $[0, \infty)$. I know that the answer is the exponential ...
user avatar
3 votes
0 answers
178 views

Thompson sampling with adaptive kernel density estimation

This is an extension to this question, which is about handling arbitrary (potentially unbounded) reward distributions for the multi-armed bandit problem. Given a sequence of observed rewards $r_t \in \...
user avatar
  • 913
3 votes
0 answers
82 views

Indirect solution for maximum entropy through sampling?

Is there a way to sample from a finite set $\{A,B,C,D\}$ such that the limiting empirical proportions converges to the maximum entropy solution of their probabilities consistent with known constraints?...
user avatar
  • 1,297
3 votes
0 answers
223 views

Maximum Entropy without the probabilities

The entropy of a distribution is defined as $H = -\sum_{i=1}^n p_i \log(p_1)$ The principle of maximum entropy states that we should choose the distribution, subject to our constraints, that ...
user avatar
  • 1,297
3 votes
0 answers
96 views

Maximal entropy distribution with given conditionals

It is well known that of all the joint distributions $p(x,y)$ with fixed marginals $p(x),p(y)$, the one with the highest entropy is: $$ p(x,y)=p(x)p(y). $$ Suppose instead that we have conditionals. ...
user avatar
  • 311
3 votes
1 answer
136 views

Maximum entropy distribution $> 0$ with vanishing probability at zero?

I know that the maximum entropy distribution if $x > 0$ and the mean is known is the exponential distribution. However, a large percentage of the probability for this distribution is close to zero (...
user avatar
  • 161
3 votes
0 answers
222 views

MaxEnt prior for positive variable that decreases monotonically to zero at a specified bound

I'm trying to devise a prior for a model parameter $x$ about which I know the following things: It is strictly positive. There is a maximum possible value $x_m$. Larger values are less likely than ...
user avatar
  • 205
2 votes
0 answers
75 views

Maximum entropy discrete distributions with specified mean

Consider a discrete distribution on {1, ...,n}, with mean given as $m$, what is the maximum entropy distribution? I know it takes the form $p_{X}(k)=ar^{k}$ and is a geometric distribution when n is ...
user avatar
2 votes
0 answers
44 views

entropy regularization in generative model

I am wondering if it is possible to use entropy as a regularization in a generative model. For example, in the conjugate model where $x_i \in X$ is observed data and generated from a Normal ...
user avatar
  • 707
2 votes
0 answers
138 views

Multiplying vector by the covariance matrix only known approximately

(cross-posted on math.SE) For random variable $(x,y)$ in $\mathbb{R}^{2d}$ and vector $v$, I need to perform the following operation on a $d \times d$ covariance matrix $E[xy']$ $$T(v)=E[xy']v$$ The ...
user avatar
2 votes
0 answers
21 views

Maximum entropy prior for r.v. supported on real line with no other constraints?

What would be a suitable maximum entropy prior for a random variable supported on the real line with no other constraints (i.e. unknown mean, unknown variance, unknown bounds)? All kinds of answers (...
user avatar
2 votes
0 answers
57 views

Determining a probability distribution from constraints on where its mass is

Let $X$ be a random variable over the real line. Suppose that we know that $X$ is a Pearson distribution. Furthermore, suppose we know how the mass of $X$ is distributed into 6 intervals, so that if $...
user avatar
2 votes
0 answers
86 views

Normal max entropy

In the following link, at the bottom of the page. https://www.dsprelated.com/freebooks/sasp/Maximum_Entropy_Property_Gaussian.html The Gaussian maximum entropy is derived. I have a question about ...
user avatar
  • 427
2 votes
0 answers
525 views

How can I maximise binary cross entropy loss?

I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
user avatar
  • 21
2 votes
0 answers
27 views

Incorporating knowledge of aggregate outcomes to constrain predictions on finer scales

I've got county-level longitudinal data on the timing of an event between the years 1998 and 2012, and I want to use it to form a predictive model for the time that that event will occur in future ...
user avatar
  • 12.3k
2 votes
0 answers
184 views

Interpreting base measure in exponential family as an improper prior (because entropy)

Long-time listener, first-time caller. I'm reading the Wikipedia pages on exponential families and maximum entropy probability distributions, and trying to wrap my head round the role of the base ...
user avatar
2 votes
0 answers
68 views

Solution to moment problem imply MaxEnt solution

I have a moment problem with arbitrary moment functions $f_i$ and a finite number of constraints, $$ \int p(x) f_i(x) dx = c_i. $$ Does the existence of an exact Pade solution of the form, $$ p(x) = \...
user avatar
  • 1,254
2 votes
0 answers
79 views

Can we use gradient desent method in maximum entropy model?

I see a lot of implementations use GIS or IIS to train the maximum entropy model. Can we use gradient desent method? If we can use it, why most tutorial directly tell GIS or IIS methos, but do not ...
user avatar
2 votes
0 answers
56 views

Is the maximum entropy distribution the same for conditional and unconditional moments?

Suppose I have a set of observations drawn from some finite interval from a distribution that has a range that includes that interval but extends beyond it. This could be either because of the values ...
user avatar
  • 2,617
2 votes
0 answers
325 views

Got an entropy-ish function for a multinomial distribution? Graph theory and Bayes net related

I have a discrete variable $X$ that can take on one of three states; $a$, $b$, and $c$. Thus it has two parameters $p_a = P(X = a)$ and $p_b = P(X = b)$, of course $P(X = c) = 1 - p_a - p_b$. I am ...
user avatar
2 votes
0 answers
70 views

How MLN and MaxEnt are different?

To me it seems that MLN (Markov logic network) and MaxEnt (Maximum Entropy classifier) solve the same formula: $$ p(y|x) = \frac{exp(\sum_i \lambda_if_i(x,y))}{\sum_y exp(\sum_i \lambda_if_i(x,y))} $$ ...
user avatar
2 votes
0 answers
160 views

Maximum Entropy with no index

This is a simpler problem than trying to solve, but have a feeling once get the methodology I can apply it to the harder problem. Let $ H(p)= -q \ln(q) - p \ln(p) $ be the entropy of the Bernoulli ...
user avatar
  • 1,297
1 vote
1 answer
49 views

How can we use shannon entropy to discriminate between two similar probability distribution function?

I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
user avatar
  • 11
1 vote
0 answers
37 views

Maximum entropy distribution of a positive continuous variable with known mean and vanishing probability at 0

I am working on a problem where I know that the variable of concern $x$ is positive, and has no upper bound on its value and whose probability would vanish as we approach 0, $\lim_{x \rightarrow 0^+} ...
user avatar
1 vote
0 answers
64 views

What is the maximum entropy joint Bernoulli distribution with fixed covariances and individual means?

We have Bernoulli variables $B_i$ with known means $E(B_i)$ and covariance matrix $\Sigma = (cov(B_i, B_j))$. What joint distribution would have the maximum entropy?
user avatar
  • 111
1 vote
0 answers
58 views

How to evaluate the likelihood of a conditional MAXENT estimation?

Suppose I have a random variable $Y$ (the outcome) and a set of random variables $\mathbf{X}$ (the input variables). I don't have access to observations of the joint distribution of $P(Y, \mathbf{X})$,...
user avatar
  • 272
1 vote
0 answers
10 views

Complexity of Maximum Entropy Algorithm in Sentiment Analysis

Does anyone know how the process of calculating the complexity of the maximum entropy algorithm and its implementation later in the sentiment analysis? Please help me, because I haven't got a ...
user avatar
1 vote
0 answers
78 views

If a zero entropy distribution implies high information a priori, what does it mean ex posteriori?

The following counteracts the statements made for the maximum entropy principle case in order to posit a pseudo "minimum entropy principle" case that is simply the polar opposite of the ...
user avatar
  • 3,139
1 vote
0 answers
21 views

Am I understanding correctly the Maximum Entropy concept using a sentence?

In the sentence "The house is white", each word carries a different amount of information. If I remove the "The" from the sentence, almost nothing happens: you are still able to ...
user avatar
1 vote
1 answer
79 views

Continuous Entropy and Maximum Entropy Solution

This is a problem that I have been working on and the mathematics of it have me fairly stumped. I am given the continuous entropy for a density $p(x)$. It is $H(X)=-\int_{0}^{\infty}p(x)\text{log}\: ...
user avatar
1 vote
1 answer
489 views

MaxEnt model vs cross entropy loss

Pardon my ignorance. I am still learning. We try to minimize the cross-entropy loss for best results. However, why should the entropy be high for a MaxEnt model for the model to be good? My ...
user avatar
  • 11
1 vote
0 answers
30 views

Why generative models in Machine Learning are Boltzmann distribution-backed?

I learned from this review paper that MaxEnt models naturally display a Boltzmann distribution for the data samples, it comes from the Principle of Maximum Entropy. But I could not understand why this ...
user avatar
  • 11
1 vote
1 answer
164 views

how to calculate entropy on matrix of words, topics

I have been digging in the concept of entropy for a while, now it comes to the implementation part I feel I am confused. Imagine that we have a matrix 20 * 3 standing for 20 words 3 topics (by 20 ...
user avatar
  • 228
1 vote
0 answers
45 views

Where is my misunderstanding/mistake in calculation about maximum entropy

I am looking through the book Philosophy of Science : Entropy and Uncertainty by Teddy Seidenfeld and have some questions. He talks about the basic dice example, where by you have a fair six sided ...
user avatar
  • 427
1 vote
0 answers
783 views

Maximum Entropy Bootstrap - how to implement?

I have a question about the Maximum Entropy Bootstrap algorithm for time series. I'm confused about how to implement step 5, where you compute the sample quantiles of the Maximum Entropy ...
user avatar
  • 121
1 vote
0 answers
83 views

How to express tail index as an expectation (for MaxEnt procedure)

I'm trying to construct a prior probability density function, $f_X(x)$, for a fat-tailed distribution using the maximum entropy (MaxEnt) method. For my known "testable" information, I have the ...
user avatar
  • 121
1 vote
0 answers
67 views

Understanding the composition law of maximum entropy

Shannon derived the principle of maximum entropy from 3 assumptions. One of those assumptions is the composition law. In other words, he derived that maximizing the probability weighted log of ...
user avatar
  • 2,179