6
$\begingroup$

I am trying to understand MLE, MAP and naive Bayes classifier, but it's difficult to understand the differences without some numerical example. Can someone give simple intuitive numerical example for each of these and how there outcomes differ from each other?

I found this example "http://www.math.utah.edu/~levin/M5080/mle.pdf" for MLE.

$\endgroup$
1
  • $\begingroup$ Check MLE and MAP $\endgroup$
    – Tim
    Jan 21, 2017 at 10:18

1 Answer 1

7
$\begingroup$

You find a coin, and want to check what are the chances (probability) that if you flip it, it would land on the "Heads" side. You flip it 10 times (a sample of 10 observations) and count the number of times if landed on "Heads", e.g. $X=3$.

You assume that what you just done is a sample from a Binomial distribution $B(10, p_0)$ - so what we're after is the parameter $p_0$ which can get values between $[0,1]$.


The likelihood function is the probability of observing $X=3$ for each possible value of $p_0$. The MLE estimator is the value that is the most likely = we could find that by the usual method of locating the point where the derivative is 0, but its easier to do that for the log-likelihood, which has a maximum at the same point since applying log is applying a monotonically increasing transformation. So log + derive + find the root of this:

$$f(p) = \binom{10}{3}p^3(1-p)^7$$

and get $p_{MLE} = 0.3$


The MAP estimator is a Bayesian estimator. These guys don't consider $p_0$ as a simple parameter, but rather as another random variable with its own distribution called the prior distribution, which is our prior belief about what the value of $p_0$ should be. In our example, I would probably be inclined to believe that $p_0 = 0.5$, as most of the coins I saw in my life were balanced (or so I thought). So let's say that our prior for p is $p \sim Normal(0.5,1)$. Our posterior probability will look like:

$$ Pr[X|p]Pr[p] = \binom{10}{3}p^3(1-p)^7\frac{1}{\sqrt{2\pi}}exp(-\frac{(p-0.5)^2}{2})$$

log, derive, find root and get $p_{MAP} = 0.304$


Naive Bayes is a classifier which gives a binary prediction. Assuming there are 2 classes of coins in the world - 95% are balanced ($p=0.5$) and 5% are rigged ($p=0.7$). Naive Bayes would use that prior distribution, apply it to your sample of $X=3$, and help you decide if its more likely that your coin is balanced or rigged. You can see it as an instantiation of the MAP estimator for the 2 classes, and selecting the more probable one. We denote balanced as $y=1$ and rigged as $y=0$, and we will predict $y=1$ if:

$$ \frac{Pr[y=1|X=3]}{Pr[y=0|X=3]} > 1 $$

More on inference here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.