As far as I studied, the Bayesian approach is the most correct in Machine Learning. Thought, I solved an exercise where it was required to find out the decision boundary in predicting two different classes, assumed to be generated with a 2D Gaussian distibution.

Here is is the Bayesian approach. The boundary is: $$C_1 \quad if \quad P(x|C_1)P(C_1) > P(x|C_2)P(C_2)$$ $$C_2 \quad else,$$ where $P(x | C_1)$ and $P(x | C_2)$ are assumed to be Gaussian (the parameter are found with max-log likelihood), $P(C_1) = N_1 / (N_1 + N_2)$ and $P(C_2) = N_2 / (N_1 + N_2)$. Bayesian

For the frequentist approach I considered the decision boundary computed as follows: $$C_1 \quad if \quad P(x|C_1) > P(x|C_2)$$ $$C_2 \quad else.$$ Frequentist Decision Boundary Frequentist

Now the problem is that, graphically speaking, the frequentist approach seems to be more precise, while the bayesian one seems to exaggerate the blue area which clearly cover an area that is dominated by red samples. How will you justify the usage of bayesian approach?

  • 3
    $\begingroup$ Can you edit your question to reflect the problem, the approaches, and how you are measuring "right"-ness? See the wiki for How to Ask a Question on CV $\endgroup$ – AdamO Jun 6 '17 at 19:50
  • $\begingroup$ Estimating $P(C_1)$ and $P(C_2)$ from the data as you do is actually frequentist. Not that there's anything wrong with that... $\endgroup$ – Lewian May 20 '20 at 9:29

Many issues with your analysis and question:

1: the premise "the Bayesian approach is the most correct in Machine Learning" is flawed.

2: It's not actually Bayesian. Base a decision on Pr(C|x) and not Pr(x|C)

3: The LR statistic is UMP for Gaussian mixtures. We expect frequentist and approximate Bayesian inference to be superior.

4: Your Bayes inference looks wrong. It looks like you set C to a single variable representing the Bernoulli proportion of class membership, presumably a beta distribution, which has simply multiplied the same likelihood ratio threshold by an arbitrary factor and led to a mis-calibrated decision rule. Each observation has a class membership, so that is N bernoulli variables, either conditional (hypergeometric) on the proportion or not (binomial) and a prior on the probabilities of class membership taken from a dirichlet distribution. The dirichlet posterior can be updated using the EM estimated centroids for the Gaussian mixture.

  • $\begingroup$ What does UMP stand for? $\endgroup$ – Matthew Drury Jun 6 '17 at 20:35
  • $\begingroup$ "Uniformly Most Powerful" $\endgroup$ – knrumsey Jun 6 '17 at 20:55
  • $\begingroup$ Regard the 2) I thought that $Pr(C|x) = P(x|C)P(C)/P(x)$ and thus $P(C_1 | x) < P(C_2 | x)$ iif $P(x|C_1)P(C_1) < P(x|C_2)P(C_2)$. Regards the 4) I had to compute $P(x|C)$ assuming it as Gaussian. Thus $\mu= \sum_i X_i / N$ and $\Sigma = \frac{1}{N-1} (\sum_i X_i - \mu)(\sum_i X_i - \mu)^T$. The reason I computed $P(x|C)$ in that way is that it was required by the assignment. Thus I don't understand what point of my bayesian inference is wrong. Could you explain it in easier words? $\endgroup$ – Sam Jun 6 '17 at 21:44
  • $\begingroup$ @Samuele can you correct subscripts so that we know what we're doing? $C_i$ should be indexed over $i=1, \ldots, n$ and Bernoulli valued with a prior $p_i$. $C_i$ is not observed but it is data so it is fixed. $\endgroup$ – AdamO Jun 7 '17 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.