Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

Filter by
Sorted by
Tagged with
-1
votes
0answers
50 views

Shannon entropy and Gini impurity are interchangeable in practice yet different? [closed]

Gini impurity, not to be confused with the Gini coefficient, is also an information theoretic measure and corresponds to Tsallis Entropy with deformation coefficient $q=2$, which in physics is ...
0
votes
0answers
11 views

If a zero entropy distribution implies high information a priori, what does it mean ex posteriori?

The following counteracts the statements made for the maximum entropy principle case in order to posit a pseudo "minimum entropy principle" case that is simply the polar opposite of the ...
0
votes
1answer
29 views

Which has minimum concentration: the uniform distribution or the maximum entropy distribution?

For a continuous random variable, the uniform distribution has high entropy because it demonstrates the greatest level uncertainty. However, this conflicts with the maximum entropy principle, which ...
1
vote
0answers
25 views

Multiplying vector by the covariance matrix only known approximately

(cross-posted on math.SE) For random variable $(x,y)$ in $\mathbb{R}^{2d}$ and vector $v$, I need to perform the following operation on a $d \times d$ covariance matrix $E[xy']$ $$T(v)=E[xy']v$$ The ...
0
votes
0answers
12 views

Different Maximum Entropy and Cross-Entropy loss?

Hi what's the difference between Maximum Entropy and Cross-Entropy loss for a sentiment classification with 2 labels (positive or negative)?
0
votes
0answers
10 views

Are discrete maximum entropy distributions uniquely determined by their energy function?

I read this somewhere but can't find a reference, I'd love to be pointed in the right direction. I'm specifically interested in discrete Boltzmann distributions with interactions at different orders, ...
2
votes
2answers
60 views

How does a distribution's differential entropy correspond to its moments?

The Gaussian distribution maximizes entropy for the following functional constraints $$E(x) = \mu$$ and $$E((x-\mu)^2) = \sigma^2$$ which are just its first and second statistical moments (true ...
0
votes
0answers
25 views

Does minimizing KL-divergence result in maximum entropy principle?

The Kullback-Leibler divergence (or relative entropy) is a measure of how a probability distribution differs from another reference probability distribution. I want to know what connection it has to ...
2
votes
1answer
41 views

What does maximizing mutual information do?

In information theory, there is something called the maximum entropy principle. Are other information measures, such as mutual information, also commonly maximized? If mutual information describes the ...
0
votes
0answers
17 views

How does the maximum entropy principle affect joint entropy, mutual information, and other info measures?

The maximum entropy principle says that we should use the probability distribution of a univariate dataset that has the highest level of entropy because it offers the lowest information. How does the ...
0
votes
0answers
19 views

Is the maximum entropy probability distribution only determined through comparison?

The maximum entropy probability distribution has entropy at least as great as that of all other members of a specified class of probability distributions (pdf's). Does that mean that the pdf with ...
1
vote
2answers
58 views

Do all random variables' probability distributions have entropy?

Entropy of probability distributions is the weighted average of the log probabilities of each observation of a random variable. Does this mean that every random variable that has a probability ...
1
vote
1answer
86 views

Why do we want a maximum entropy distribution, if it has the lowest information?

It is said that the distribution with the largest entropy should be chosen as the least-informative default. That is, we should choose the distribution that maximizes entropy because it has the lowest ...
1
vote
0answers
20 views

Am I understanding correctly the Maximum Entropy concept using a sentence?

In the sentence "The house is white", each word carries a different amount of information. If I remove the "The" from the sentence, almost nothing happens: you are still able to ...
1
vote
1answer
24 views

Continuous Entropy and Maximum Entropy Solution

This is a problem that I have been working on and the mathematics of it have me fairly stumped. I am given the continuous entropy for a density $p(x)$. It is $H(X)=-\int_{0}^{\infty}p(x)\text{log}\: ...
2
votes
1answer
21 views

Is this implementation detail for solving maximum entropy on a computer correct?

I am currently looking at a paper by Mattos and Veiga, who describe an approach to solving the maximum entropy problem subject to linear constraints: $$\begin{aligned} \max_{p_i} -\sum_{i=1}^N p_i \...
1
vote
1answer
52 views

When is uniform distribution have maximum entropy instead of normal distribution?

As far as I know, when we have just data and no constraints (other than probabilities must add up to 1), the distribution that gives maximum entropy is uniform distribution. But when we know mean and ...
4
votes
0answers
46 views

Why are $\mathbb{E}( \ln(x))$ and $\mathbb{E} ( \ln(1 - x))$ reasonable descriptions of knowledge about a beta distribution?

The max entropy philosophy states that given some constraints on the prior, we should choose the prior that is maximum entropy subject to those constraints. I know that the Beta($\alpha, \beta$) is ...
0
votes
0answers
11 views

How do you use MaxEnt for Bayesan updating given new data on a toy problem about a biased coin

The article "Updating, supposing, and maxent" (paywalled) says, Bayes' rule is a special case of MAXENT. Let the random variable $I_c$ be the indicator function which takes the value 1 in the set $...
0
votes
0answers
21 views

Is there a connection between the binomial pmf and the formula for entropy? [duplicate]

Many times when two formulas "look" the same, there is some interesting mathematical result linking them. Both the log binomial likelihood and the entropy formula kind of "look" the same, in that they ...
1
vote
0answers
14 views

Do Lévy α-stable distributions maximize entropy subject to a simple constraint?

Is there a simple constraint on real-valued distributions such that the maximum entropy distribution is Lévy α-stable? Special cases include the Normal and Cauchy distributions for which the answer is ...
0
votes
0answers
21 views

Four different ways to deal with the log-likelihood of a probability density function (Python code included)

This is not really a question but more of a discussion. Please correct me where wrong and share your thoughts and past experience with regards to computing the likelihoods for continuous data models. ...
1
vote
2answers
128 views

How do I prove conditional entropy is a good measure of information?

This question is a follow-up of Does "expected entropy" make sense?, which you don't have to read as I'll reproduce the relevant parts. Let's begin with the statement of the problem A student has ...
3
votes
0answers
66 views

Are there nonparametric generative models for datasets?

Typically when I see generative models, e.g., Latent Dirichlet Allocation (JMLR) or Linear/Quadratic Discriminant Analysis (wikipedia LDA), they are probabilistic models that belong to the exponential ...
3
votes
1answer
105 views

Intuition for the uniform distribution having the maximum entropy

I saw the following explanation for Entropy in probability: (Entropy). The surprise of learning that an event with probability $p$ happened is defined as $\log_2(1/p)$, measured in a unit called ...
0
votes
1answer
101 views

MaxEnt model vs cross entropy loss

Pardon my ignorance. I am still learning. We try to minimize the cross-entropy loss for best results. However, why should the entropy be high for a MaxEnt model for the model to be good? My ...
0
votes
0answers
27 views

Maximum entropy function, with f(0)=0?

I want to derive the Maximum Entropy distribution (f(x)) with the following constraints: 1. non-negative 2. specified mean 3. specified variance 4. f(0)=0 I know how to derive the MaxEnt distro with ...
1
vote
0answers
16 views

Maximum entropy prior for dichotomous variables [closed]

I have a set of dichotomous variables $A, B, C,$... and I know their probabilities $P(A), P(B), P(C),$... as well es their pairwise dependencies $P(A \cap B), P(A \cap C), P(B \cap C),$... . Or in ...
1
vote
0answers
26 views

Why generative models in Machine Learning are Boltzmann distribution-backed?

I learned from this review paper that MaxEnt models naturally display a Boltzmann distribution for the data samples, it comes from the Principle of Maximum Entropy. But I could not understand why this ...
4
votes
0answers
39 views

How can a probability densitiy be estimated based on the maximum entropy principle, with constraints in the order statistics?

Let's say we are given a sample $\{ z_i \}$, $i=1,2,\ldots,N$, such that each value $z_i$ corresponds to the minimum of $n$ random variables $x$, i.e., $z = \min \{ x_1, x_2,\ldots,x_n \}$. The ...
0
votes
0answers
40 views

How to minimise and expectation with respect to a parameter?

Suppose that $X$ is a random variable with distribution $G$. Let $H(X;\theta)$ be a parametric function with $\theta \in \Theta \subset {\mathbb R}^p$. I want to maximize the function $$\varphi(\...
2
votes
0answers
18 views

Maximum entropy prior for r.v. supported on real line with no other constraints?

What would be a suitable maximum entropy prior for a random variable supported on the real line with no other constraints (i.e. unknown mean, unknown variance, unknown bounds)? All kinds of answers (...
1
vote
1answer
62 views

how to calculate entropy on matrix of words, topics

I have been digging in the concept of entropy for a while, now it comes to the implementation part I feel I am confused. Imagine that we have a matrix 20 * 3 standing for 20 words 3 topics (by 20 ...
2
votes
0answers
37 views

Determining a probability distribution from constraints on where its mass is

Let $X$ be a random variable over the real line. Suppose that we know that $X$ is a Pearson distribution. Furthermore, suppose we know how the mass of $X$ is distributed into 6 intervals, so that if $...
2
votes
0answers
47 views

Maximum entropy probability distribution over non-negative support and finite mean?

I'm trying to derive which univariate probability distribution maximizes entropy, assuming finite mean $\mu$ and non-negative support $[0, \infty)$. I know that the answer is the exponential ...
2
votes
2answers
448 views

Proving that Shannon entropy is maximised for the uniform distribution

I know that Shannon entropy is defined as $-\sum_{i=1}^kp_i\log(p_i)$. For the uniform distribution, $p_i=\frac{1}{k}$, so this becomes $-\sum_{i=1}^k\frac{1}{k}\log\left(\frac{1}{k}\right)$. Further ...
0
votes
1answer
109 views

Estimating maximum entropy distribution given first n moments

Is there a good way to estimate the pdf, pdf up to a constant multiple, cdf, or quantile function of a distribution given the first n moments? A closed form for one of those functions in terms of the ...
1
vote
1answer
83 views

Is the marginal distribution of a maximum entropy distribution also a maximum entropy distribution?

I am considering under what circumstances maximum entropy distributions are closed under marginalization. The main case I am interested is the following setting: a finite set of discrete random ...
4
votes
1answer
84 views

Maximum entropy distribution on the hypercube

Given the first two moments, the maximum entropy distribution over $\mathbb{R}$ is known the be the normal distribution. What is the analogue for a distribution over $[0,1]$ given either only the ...
2
votes
1answer
127 views

Same maximum entropy and measures [closed]

As the author Christian Robert asks about reference measures that are absolutely continuous to one another, and from what I can gather this just means they have the same null set? But would that not ...
2
votes
0answers
79 views

Normal max entropy

In the following link, at the bottom of the page. https://www.dsprelated.com/freebooks/sasp/Maximum_Entropy_Property_Gaussian.html The Gaussian maximum entropy is derived. I have a question about ...
3
votes
1answer
229 views

Some Questions about reference measures and maximum entropy priors (from The Bayesian Choice)

I am relatively new to statistics and Bayesian theory but I am trying to understand it by working through a few books. There are some things I am confused about. ( As I believe my analysis and such is ...
1
vote
0answers
40 views

Where is my misunderstanding/mistake in calculation about maximum entropy

I am looking through the book Philosophy of Science : Entropy and Uncertainty by Teddy Seidenfeld and have some questions. He talks about the basic dice example, where by you have a fair six sided ...
13
votes
1answer
247 views

Maximum Entropy with bounded constraints

Assume we have the problem of estimating the probabilities $\{p_1,p_2,p_3\}$ subject to: $$0 \le p_1 \le .5$$ $$0.2 \le p_2 \le .6$$ $$0.3 \le p_3 \le .4$$ with only the natural constraint of $p_1+...
2
votes
0answers
329 views

How can I maximise binary cross entropy loss?

I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
2
votes
0answers
24 views

Incorporating knowledge of aggregate outcomes to constrain predictions on finer scales

I've got county-level longitudinal data on the timing of an event between the years 1998 and 2012, and I want to use it to form a predictive model for the time that that event will occur in future ...
1
vote
1answer
213 views

What is the maximum entropy distribution given the median (instead of the mean)?

Given that the median seems to be a more robust statistic than the mean/average, I was wondering if there is a solution of the maximum entropy distribution given the median (or the median and some ...
4
votes
1answer
410 views

Maximum Entropy: another name for Maximum Likelihood or a legit Bayes procedure?

In some more recent works (like this and this for instance), MaxEnt is unequivocally bound to Maximum Likelihood (ergo Classical Inference). In some other older works (like Jaynes article), MaxEnt ...
0
votes
0answers
172 views

Computational complexity of MaxiEnt classifier

I know that the time complexity of logistic regression can be as low as linear when the optimizer/solver is assumed to be linear, such as L-BFGS (this link) I know that multinomial logistic regression ...
1
vote
0answers
532 views

Maximum Entropy Bootstrap - how to implement?

I have a question about the Maximum Entropy Bootstrap algorithm for time series. I'm confused about how to implement step 5, where you compute the sample quantiles of the Maximum Entropy ...