8
$\begingroup$

I am reading E. T. Jaynes' probability theory book, and I am at chapter 11 where he introduces the maximum entropy principle. I understand that Jaynes separates the notion of probability from that of frequency, which I like. But his arguments leading to the maximum entropy principle seem to mix up these two concepts:

Following the example in the book at section 11.5, let's say that we have a 3 sided "die", which we have thrown for an unknown number of times and we only know the sample mean of the results, called $\overline{m}$. We want to assign a probability to each of the 3 sides of the die. Jaynes reasons that, given only this information, the most reasonable way to assign these probabilities is to choose the most uncertain probability distribution that satisfies all the requirements (i.e. observations, in this case the sample mean). Now, this is quite arbitrary, but I accept it as an intuitively compelling method.

Then, Jaynes reasons that, because we know the sample mean turned out to be $\overline{m}$, this is a constraint for our assigned probability distribution's mean to also be $\overline{m}$. This is where he loses me. If I understand correctly, the sample mean is only a feature of the frequency distribution, and cannot be used to rule out any probability distribution, because, except for some edge cases, any sample mean is possible (i.e. probability of larger than zero) for any probability distribution. So, probability distributions that do not have mean $\overline{m}$ are also compatible with the observation that the sample mean equals $\overline{m}$ and should be considered in the optimization process (and in this case, if we accept my argument, we have to assign a uniform probability distribution, regardless of the value of $\overline{m}$, as opposed to Jaynes' result).

Is there anything I am missing?

Edit: Please note that, when I say I accept this method, I mean that I accept that using the most uncertain probability distribution that satisfies constraints is a good measure for assigning probabilities. The part of Jaynes' reasoning that I do not accept is that sample mean is a constraint for the probability mean (as I reasoned above). The reason I do not accept it is that this seems to contradict the whole philosophy that the previous chapters were built upon, which is frequency$\neq$probability in the general case.

$\endgroup$
2
  • 2
    $\begingroup$ I love Jaynes' work and have read most of his papers as well as the book, but to be honest I think the justification for is the weakest part of his work - I don't think he ever really had a compelling reason why we should take the constraints to be expectations on the mean, other than "this is what leads to all the really cool results." That's a perfectly respectable reason in itself of course, but given Jaynes' otherwise very solid philosophical arguments I always found it odd that this was lacking. $\endgroup$
    – N. Virgo
    Jan 3, 2023 at 9:59
  • $\begingroup$ @N.Virgo I think one could use any number of moments obtained from the data. How many depends on what in ML is referred to as bias-variance tradeoff - too many moments result in overfitting and make results less genetalizable to further observations. Jaynes himself was peobably guided by the analogy with statistical physics. $\endgroup$
    – Roger V.
    Jan 4, 2023 at 5:56

2 Answers 2

3
$\begingroup$

Jaynes' reasoning here is essentially Bayesian, where the probability is a measure of uncertainty of our knowledge about something. This is opposed to frequentist reasoning, which is based on the assumption that one could potentially know the whole population, its parameters, and the underlying probability distribution (such knowledge is often impossible - e.g., the population can be infinite, or we might be dealing with a series events that may continue far into the future - like sunrise.)

Thus, Jaynes bases his reasoning on the knowledge available to him (such as sample average $\bar{m}$), but does not make assumptions about what the "true" population average $\mu$ is or whether such a population average even exists.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer. I am aware of Jaynes' Bayesian view, and in fact it is exactly why I am confused because his argument does not seem to follow the Bayesian definition of probability and seems to be of the type that he explicitly criticizes in the previous chapters (i.e. making assumptions about the connection between frequency and probability without any rigorous mathematical derivation of any such connection) $\endgroup$
    – Feri
    Jan 3, 2023 at 21:36
  • $\begingroup$ @Feri sample mean is calculated from the observed data - it does not require any assumptions about ppopulation mean or even the existence thereof. Reducing uncertainty using available data is the essence of the Bayesian approach. $\endgroup$
    – Roger V.
    Jan 4, 2023 at 6:02
2
$\begingroup$

Jaynes reasons that, given only this information, the most reasonable way to assign these probabilities is to choose the most uncertain probability distribution that satisfies all the requirements (i.e. observations, in this case the sample mean). Now, this is quite arbitrary, but I accept it as an intuitively compelling method.

You say that you accept this method, but then you immediately reject it in the next paragraph. Of course it is true that the expected value of the probability distribution need not match the sample mean, but the whole content of the method just described (which you at first say you accept) is that he is matching the distribution moments to the sample moments in his constraints and then seeking the most uncertain distribution under those constraints. If you don't use this moment constraint, and just use a uniform distribution, then you are not using the method he just described. What is it about his method that you imagine you accept if not that?

$\endgroup$
1
  • $\begingroup$ Thank you for your answer. I added an edit to the question to address the confusion. $\endgroup$
    – Feri
    Jan 3, 2023 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.