1
$\begingroup$

I am reading some Mahchine learning lecture notes, and the writer is introducing Maximum Likelihood (ML) method of parameter estimation of $\theta$ as $\text{argmax}_{\theta}Pr_{\theta}(x|H)$ where $H$ is a hypothesis and $Pr(x|H)$ is the likelihood of the observation.

Now later on, when he introduced the mixture of Gaussians, He used Bayes theorem to define the probability of a class $c_k$ as $Pr(c_k|x)=kPr(x|c_k)Pr(c_k)$ where the probability of each class $Pr(c_k)$ is multinomial, and $Pr(x|c_k)$ is Gaussian.

Now here is where I am getting confused When it came to applying ML to learn the parameters of the posterior, namely $\pi_k=Pr(c_k)$ and the covariance matrix $\Sigma$ and means $\mu_1, ..., \mu_k$ for each distribution, We'll call all these parameters $\Theta$, then he tried to maximize:

$$Pr(X|c_k)Pr(c_k)= \Pi_n \Pi_j [\pi_jN(x_n|\pi_j,\Sigma)]^{y_j}$$

Where $y_j$ is $1$ is $x_n$ belongs to class $j$ and $0$ otherwise.

However, shouldn't he have maximized just $Pr(X|c_k)$ which is the likelihood here?

$\endgroup$

1 Answer 1

1
$\begingroup$

If the author's intent was to find the maximum likelihood estimate of the parameters $\Theta$, then yes, the relevant optimization problem is $$ \max_\Theta \text{Pr}(X | c_k). $$ But it looks like what's going on here is actually an example of maximum a posteriori (MAP) estimation, which produces an estimate of the parameters by maximizing the posterior distribution $\text{Pr}(c_k|X)$. As you point out, the posterior distribution is proportional to the product of the likelihood and the prior distribution, $\text{Pr}(X|c_k)\text{Pr}(c_k)$, so it suffices to find the value of $\Theta$ that maximizes this product.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.