Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

2
votes
0answers
28 views

Determining a probability distribution from constraints on where its mass is

Let $X$ be a random variable over the real line. Suppose that we know that $X$ is a Pearson distribution. Furthermore, suppose we know how the mass of $X$ is distributed into 6 intervals, so that if $...
2
votes
0answers
21 views

Maximum entropy probability distribution over non-negative support and finite mean?

I'm trying to derive which univariate probability distribution maximizes entropy, assuming finite mean $\mu$ and non-negative support $[0, \infty)$. I know that the answer is the exponential ...
1
vote
1answer
19 views

Proving that Shannon entropy is maximised for the uniform distribution

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further ...
0
votes
0answers
19 views

Can a Jeffreys prior be used as an Information maximizing distribution if Information is defined using differential entropy?

I know that a Jeffreys prior is the information maximizing distribution for the statistical channel. However, I want to know if I define mutual information as $$I(x;y)=h(x)-h(x|y)$$ where $h(.)$ is ...
0
votes
0answers
24 views

Are there connections between Maximum Entropy and Variational Inference?

I would like to ask what if there are any connections between Maximum Entropy and Variational Inference? I suspect that they are related somehow.
0
votes
1answer
30 views

Estimating maximum entropy distribution given first n moments

Is there a good way to estimate the pdf, pdf up to a constant multiple, cdf, or quantile function of a distribution given the first n moments? A closed form for one of those functions in terms of the ...
1
vote
1answer
41 views

Is the marginal distribution of a maximum entropy distribution also a maximum entropy distribution?

I am considering under what circumstances maximum entropy distributions are closed under marginalization. The main case I am interested is the following setting: a finite set of discrete random ...
2
votes
0answers
50 views

Maximum entropy distribution on the hypercube

Given the first two moments, the maximum entropy distribution over $\mathbb{R}$ is known the be the normal distribution. What is the analogue for a distribution over $[0,1]$ given either only the ...
1
vote
1answer
104 views

Same maximum entropy and measures [closed]

As the author Christian Robert asks about reference measures that are absolutely continuous to one another, and from what I can gather this just means they have the same null set? But would that not ...
1
vote
0answers
66 views

Normal max entropy

In the following link, at the bottom of the page. https://www.dsprelated.com/freebooks/sasp/Maximum_Entropy_Property_Gaussian.html The Gaussian maximum entropy is derived. I have a question about ...
3
votes
1answer
195 views

Some Questions about reference measures and maximum entropy priors (from The Bayesian Choice)

I am relatively new to statistics and Bayesian theory but I am trying to understand it by working through a few books. There are some things I am confused about. ( As I believe my analysis and such is ...
0
votes
0answers
34 views

Where is my misunderstanding/mistake in calculation about maximum entropy

I am looking through the book Philosophy of Science : Entropy and Uncertainty by Teddy Seidenfeld and have some questions. He talks about the basic dice example, where by you have a fair six sided ...
0
votes
0answers
22 views

Confusion about maximising vs minimising (relative) entropy terminology and methods

I'm studying rules of inference for updating from a prior probability distribution to a posterior. One method for doing this is by maximising entropy, subject to constraints. I'm reading papers like ...
0
votes
0answers
41 views

How to interpret log-likelihood score as compared to mse

Say one has a linear dynamic system as follows: $x_k = Fx_{k-1} + v_k$ $y_k = Hx_{k-1} + w_k$ with $v \sim (0, Q)$ and $w \sim (0, R)$. I am estimating $(x)_k$ using a normal Kalman Filter and ...
10
votes
0answers
167 views

Maximum Entropy with bounded constraints

Assume we have the problem of estimating the probabilities $\{p_1,p_2,p_3\}$ subject to: $$0 \le p_1 \le .5$$ $$0.2 \le p_2 \le .6$$ $$0.3 \le p_3 \le .4$$ with only the natural constraint of $p_1+...
1
vote
0answers
134 views

How can I maximise binary cross entropy loss?

I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
2
votes
0answers
22 views

Incorporating knowledge of aggregate outcomes to constrain predictions on finer scales

I've got county-level longitudinal data on the timing of an event between the years 1998 and 2012, and I want to use it to form a predictive model for the time that that event will occur in future ...
1
vote
1answer
84 views

What is the maximum entropy distribution given the median (instead of the mean)?

Given that the median seems to be a more robust statistic than the mean/average, I was wondering if there is a solution of the maximum entropy distribution given the median (or the median and some ...
4
votes
1answer
177 views

Maximum Entropy: another name for Maximum Likelihood or a legit Bayes procedure?

In some more recent works (like this and this for instance), MaxEnt is unequivocally bound to Maximum Likelihood (ergo Classical Inference). In some other older works (like Jaynes article), MaxEnt ...
0
votes
0answers
105 views

Computational complexity of MaxiEnt classifier

I know that the time complexity of logistic regression can be as low as linear when the optimizer/solver is assumed to be linear, such as L-BFGS (this link) I know that multinomial logistic regression ...
0
votes
0answers
224 views

Maximum Entropy Bootstrap - how to implement?

I have a question about the Maximum Entropy Bootstrap algorithm for time series. I'm confused about how to implement step 5, where you compute the sample quantiles of the Maximum Entropy ...
3
votes
0answers
113 views

Thompson sampling with adaptive kernel density estimation

This is an extension to this question, which is about handling arbitrary (potentially unbounded) reward distributions for the multi-armed bandit problem. Given a sequence of observed rewards $r_t \in \...
1
vote
0answers
28 views

Why do we care about maximum entropy? [duplicate]

One justification for the ubiquity of the (multivariate) normal distribution in statistical/machine learning modeling is that it maximizes entropy among distributions with mean $\mu$ and variance ...
1
vote
0answers
74 views

How to express tail index as an expectation (for MaxEnt procedure)

I'm trying to construct a prior probability density function, $f_X(x)$, for a fat-tailed distribution using the maximum entropy (MaxEnt) method. For my known "testable" information, I have the ...
0
votes
0answers
54 views

Entropy of a factorised joint distribution

Suppose I have three discrete random variables $X, Y$ and $Z$. Their joint distribution factorises as so: $$ P(X,Y,Z) = P(X)P(Y)P(Z) $$ i.e. they are fully independent variables. Now suppose I want ...
1
vote
1answer
171 views

Entropy of a matrix with Bernoulli distributed (binary entries) row-vectors

The entropy $H[x]$ of a Bernoulli distributed binary random variable $x$ is given by : $$ H[x]=−θlnθ−(1−θ)ln(1−θ) $$ where $$ p(x=1∣θ)= \theta \\ p(x=0∣θ)=1−θ $$ Now, suppose I have a vector as so: ...
8
votes
1answer
147 views

Support vector machines (SVMs) are the zero temperature limit of logistic regression?

I was had a quick discussion recently with a knowledgeable friend who mentioned that SVMs are the zero temperature limit of logistic regression. The rationale involved marginal polytopes and fenchel ...
1
vote
0answers
35 views

Understanding the composition law of maximum entropy

Shannon derived the principle of maximum entropy from 3 assumptions. One of those assumptions is the composition law. In other words, he derived that maximizing the probability wieghted log of ...
0
votes
0answers
73 views

Is cross-entropy loss for MaxEnt models convex? If so, how do I prove it?

I am trying to prove if cross-entropy loss for MaxEnt models is convex. My first attempt at approaching this problem is to compute the Hessian matrix of second-order partial derivatives and showing ...
2
votes
1answer
58 views

Bimodal MaxEnt distributions?

What kind of constrains give rise to bimodal distributions in the Maximum Entropy formalism? Are there any known results in this topic?
4
votes
1answer
75 views

Maximum Entropy modelling - likelihood equation

I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be ...
1
vote
1answer
68 views

Measure non-randomness of numeric license plates

License plates for cars in Switzerland have the 2-letter abbreviation from the Canton and then between 1 and 6 numeric digits. There are no alphabet characters in the license plate, and therefore no ...
1
vote
0answers
67 views

Sampling MaxEnt Distribution with PyMC3

I’m trying to use PyMC3 to sample from a maximum entropy distribution over binary patterns of length $N$, constrained by means and pairwise correlations. The ...
4
votes
0answers
50 views

Distribution (CDF) estimation for strictly increasing, continuous distribution with compact support

For all $t\in 1,\dots,T$, suppose $x_t\in [0,1]$ is a draw from a distribution with unknown CDF $F:[0,1]\rightarrow [0,1]$. For future use, define $\tilde{x}\in [0,1]^T$ to be a vector containing $x_1,...
0
votes
1answer
105 views

MaxEnt with given variance (but mean unknown)

The MaxEnt distribution of $x \in (-\infty,\infty)$ with given mean and variance is the Gaussian. What happens when the mean is unknown? What is the MaxEnt distribution of $x\in(-\infty,\infty)$ with ...
1
vote
0answers
40 views

MaxEnt distribution of $x\in(-\infty,\infty)$ with given mean?

The MaxEnt distribution of $x\in[0,\infty)$ with given mean is the exponential. The MaxEnt distribution of $x \in (-\infty,\infty)$ with given mean and variance is the Gaussian. Is there a MaxEnt ...
4
votes
2answers
152 views

Random values in fixed interval - how to assign probability distribution?

Please excuse my lack of terminology. I am just a humble discrete optimizer Assume we have a kind of "coin toss" where the result is not binary, but a rational number between -1 and 1. There is ...
2
votes
0answers
72 views

Interpreting base measure in exponential family as an improper prior (because entropy)

Long-time listener, first-time caller. I'm reading the Wikipedia pages on exponential families and maximum entropy probability distributions, and trying to wrap my head round the role of the base ...
0
votes
1answer
95 views

Regularization versus feature reduction in species distribution modeling using Maxent

I am wondering if there is a need to set the beta multiplier in Maxent (species distriubition modeling approach) if one is also reducing features using a contribution threshold. I have seen a number ...
2
votes
0answers
51 views

Solution to moment problem imply MaxEnt solution

I have a moment problem with arbitrary moment functions $f_i$ and a finite number of constraints, $$ \int p(x) f_i(x) dx = c_i. $$ Does the existence of an exact Pade solution of the form, $$ p(x) = \...
3
votes
0answers
125 views

The intuition of maximum entropy [duplicate]

I see a lot of implementation of maximum entropy. Can someone explain why it is possible to estimate the parameter based on maximum entropy?
1
vote
0answers
113 views

What is the minimum (differential) entropy at a given variance?

Given a set of samples $S$ from an unknown multidimensional real-valued distribution, I use the multivariate normal distribution $N(\mu(S),\Sigma(S))$ to compute its upper entropy limits. This is ...
1
vote
1answer
66 views

Clarifying Berger et al. “A Maximum Entropy Approach to NLP”

Unfortunately, I do not know where else to turn to for clarifications that I require regarding the notations and definitions in the initial sections of the paper by Berger et al (1996). In Section 3....
2
votes
1answer
702 views

What is the best way to perform sentence segmentation for textual analysis?

I am working on textual dataset containing data from official documents like reports by companies, legal documents, speeches by directors to shareholders etc. The content of textual documents are like:...
1
vote
0answers
12 views

Does it making sense to change the maximum entropy condition in MaxEnt or CRFs according to distribution?

Assuming English words are in Pareto (Zipf) distribution, when applying a log-linear model, should it still apply Gibbs measure of the maximum entropy condition, or one can change it to the Pareto ...
2
votes
1answer
130 views

Understanding parameter learning use of Principle of maximum entropy in Bayesian networks

I was reading Bayesian network on wiki: https://en.wikipedia.org/wiki/Bayesian_network And It stated for parameter learning use of "Principle_of_maximum_entropy"....
2
votes
1answer
39 views

Reference request for Boltzmann's Theorem on maximum entropy distributions

Wikipedia (https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#Distributions_with_measured_constants) describes in detail a theorem which gives the maximum entropy probability ...
1
vote
0answers
50 views

“Entropy” in Fantasy Football League

Compare playoff structures of two leagues. One (League A) in which we add divisions (where the top finishers in a division move on to the playoffs based on overall-record) and the other (League B) ...
6
votes
1answer
2k views

When does the maximum likelihood correspond to a reference prior?

I have been reading James V. Stone's very nice books "Bayes' Rule" and "Information Theory". I want to know which sections of the books I did not understand and thus need to re-read further. The ...
4
votes
1answer
148 views

What is the maximum entropy distribution given values for several quantiles of one sample?

Suppose I know the value of n specific quantiles from a large sample. What is the maximum entropy distribution over the real line given those values?