0
$\begingroup$

As far as I studied, the Bayesian approach is the most correct in Machine Learning. Thought, I solved an exercise where it was required to find out the decision boundary in predicting two different classes, assumed to be generated with a 2D Gaussian distibution.

Here is is the Bayesian approach. The boundary is: $$C_1 \quad if \quad P(x|C_1)P(C_1) > P(x|C_2)P(C_2)$$ $$C_2 \quad else,$$ where $P(x | C_1)$ and $P(x | C_2)$ are assumed to be Gaussian (the parameter are found with max-log likelihood), $P(C_1) = N_1 / (N_1 + N_2)$ and $P(C_2) = N_2 / (N_1 + N_2)$. Bayesian

For the frequentist approach I considered the decision boundary computed as follows: $$C_1 \quad if \quad P(x|C_1) > P(x|C_2)$$ $$C_2 \quad else.$$ Frequentist Decision Boundary Frequentist

Now the problem is that, graphically speaking, the frequentist approach seems to be more precise, while the bayesian one seems to exaggerate the blue area which clearly cover an area that is dominated by red samples. How will you justify the usage of bayesian approach?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Can you edit your question to reflect the problem, the approaches, and how you are measuring "right"-ness? See the wiki for How to Ask a Question on CV $\endgroup$
    – AdamO
    Commented Jun 6, 2017 at 19:50
  • $\begingroup$ Estimating $P(C_1)$ and $P(C_2)$ from the data as you do is actually frequentist. Not that there's anything wrong with that... $\endgroup$
    – Christian Hennig
    Commented May 20, 2020 at 9:29

1 Answer 1

Reset to default
1
$\begingroup$

Many issues with your analysis and question:

1: the premise "the Bayesian approach is the most correct in Machine Learning" is flawed.

2: It's not actually Bayesian. Base a decision on Pr(C|x) and not Pr(x|C)

3: The LR statistic is UMP for Gaussian mixtures. We expect frequentist and approximate Bayesian inference to be superior.

4: Your Bayes inference looks wrong. It looks like you set C to a single variable representing the Bernoulli proportion of class membership, presumably a beta distribution, which has simply multiplied the same likelihood ratio threshold by an arbitrary factor and led to a mis-calibrated decision rule. Each observation has a class membership, so that is N bernoulli variables, either conditional (hypergeometric) on the proportion or not (binomial) and a prior on the probabilities of class membership taken from a dirichlet distribution. The dirichlet posterior can be updated using the EM estimated centroids for the Gaussian mixture.

Improve this answer
$\endgroup$
4
  • $\begingroup$ What does UMP stand for? $\endgroup$
    – Matthew Drury
    Commented Jun 6, 2017 at 20:35
  • $\begingroup$ "Uniformly Most Powerful" $\endgroup$
    – knrumsey
    Commented Jun 6, 2017 at 20:55
  • $\begingroup$ Regard the 2) I thought that $Pr(C|x) = P(x|C)P(C)/P(x)$ and thus $P(C_1 | x) < P(C_2 | x)$ iif $P(x|C_1)P(C_1) < P(x|C_2)P(C_2)$. Regards the 4) I had to compute $P(x|C)$ assuming it as Gaussian. Thus $\mu= \sum_i X_i / N$ and $\Sigma = \frac{1}{N-1} (\sum_i X_i - \mu)(\sum_i X_i - \mu)^T$. The reason I computed $P(x|C)$ in that way is that it was required by the assignment. Thus I don't understand what point of my bayesian inference is wrong. Could you explain it in easier words? $\endgroup$
    – Sam
    Commented Jun 6, 2017 at 21:44
  • $\begingroup$ @Samuele can you correct subscripts so that we know what we're doing? $C_i$ should be indexed over $i=1, \ldots, n$ and Bernoulli valued with a prior $p_i$. $C_i$ is not observed but it is data so it is fixed. $\endgroup$
    – AdamO
    Commented Jun 7, 2017 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.