Suppose you tossed a coin multiple times. Sometimes you got heads and other times you got tails. You recorded your experiment in a dataset $ X$. Now you want to estimate the parameter θ (which represents the probability of a coin landing heads) of the Bernoulli distribution using probabilistic modeling. You want to do MAP (Maximum A Posteriori) estimation of parameter $\theta $ where the likelihood is the Bernoulli distribution and the prior is the Gaussian distribution.

Posterior distribution: $P(\theta\mid X)=P(X\mid \theta)\times P(\theta) / P(X)$ where $X={x_1,x_2,\ldots,x_n}$

Likelihood: $P(X\mid \theta)=P(x_1\mid \theta)\times P(x_2\mid\theta)\ldots\times P(x_n\mid\theta)$ because the realizations are i.i.d and $P(x_i\mid\theta)$ is the Bernoulli distribution

Prior: $P(\theta)$ is the Gaussian distribution with the parameters $\mu$ as the mean and $\sigma$ as the standard deviation.

For doing the MAP estimation, I have the following questions:

  1. knowing that $\theta$ is conditioned: $0 \leq \theta \leq 1$ (because it is a probability), does this mean that we have to use the Lagrange multiplier method (Please answer this question before moving to the next one)?

  2. Knowing that σ and μ are connected with a formula. $\sigma=(E(X^2)−\mu^2)^{1/2}$ (for the continuous case), does this mean that we have to use the Lagrange multiplier method? if yes, then how will this will be calculated in the continuous case (i.e. I tried to calculate $\sigma$ but it is a complicated task in the continuous case and I do not know if it is even possible)

  3. Could you clarify how to do the MAP estimation in this case (kindly clarify it mathematically if possible)

I am interested in the answers for all these three points, but an answer to any of these questions are also welcome!

P.S. I am a computer science engineer (i.e. statistics is not my domain), so please be patient with me.

  • 5
    $\begingroup$ Note that a Gaussian prior is inappropriate, as $\theta \in (0,1)$ but the support of the Gaussian distribution is $(-\infty, \infty)$. $\endgroup$
    – jbowman
    Commented Aug 6, 2023 at 21:19
  • 1
    $\begingroup$ You take the product of the prior density and the likelihood to give something proportional to the density of the posterior distribution. If you insist on the MAP (there is no loss function for which it is the best choice as a point estimate from a continuous posterior distribution) then you want the mode of the posterior distribution, which you can find as the value of $\theta$ which maximises the product of the prior density and the likelihood. There should be no need for a Lagrange multiplier. $\endgroup$
    – Henry
    Commented Aug 6, 2023 at 22:17
  • 1
    $\begingroup$ @MosabShaheen There are many misconceptions in what you have written. A loss function here is the cost of errors in your point estimate. There is no relationship between the parameters $\mu$ and $\sigma$ in a Gaussian distribution (the only requirement is that $\sigma>0$ which is unrelated to $\mu$). Your prior should be specified at the start, without unknown parameters. Since $0\le \theta\le 1$, that interval or a subset of it should be the support of your prior distribution, but a Gaussian distribution has support on the whole of $\mathbb R$ $\endgroup$
    – Henry
    Commented Aug 6, 2023 at 23:59
  • 3
    $\begingroup$ 1. The parameters of the prior distribution - $\mu$ and $\sigma$ - are specified, by you, in advance, hence the word "prior". They are not calculated from the data. 2. I do not know what you mean by "there will always be a constraint between them"; there won't be. 3. I don't understand why you think Lagrange multipliers play any role in this. There is no closed-form solution to the MAP for the probability parameter of a Binomial distribution with a Gaussian prior on $p$; you will have to find it numerically, and you can just specify a lower and upper bound to the solver at that time. $\endgroup$
    – jbowman
    Commented Aug 7, 2023 at 1:39
  • 2
    $\begingroup$ Your concern is exactly the same as wondering if you need to use Lagrange multipliers to ensure that 1 + 1 = 2, because 1 + 1 are constrained to = 2, but what if they don't? $\endgroup$
    – jbowman
    Commented Aug 7, 2023 at 14:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.