Bayesian vs Frequentist example

Question

As far as I studied, the Bayesian approach is the most correct in Machine Learning. Thought, I solved an exercise where it was required to find out the decision boundary in predicting two different classes, assumed to be generated with a 2D Gaussian distibution.

Here is is the Bayesian approach. The boundary is: $$C_1 \quad if \quad P(x|C_1)P(C_1) > P(x|C_2)P(C_2)$$ $$C_2 \quad else,$$ where $P(x | C_1)$ and $P(x | C_2)$ are assumed to be Gaussian (the parameter are found with max-log likelihood), $P(C_1) = N_1 / (N_1 + N_2)$ and $P(C_2) = N_2 / (N_1 + N_2)$.

For the frequentist approach I considered the decision boundary computed as follows: $$C_1 \quad if \quad P(x|C_1) > P(x|C_2)$$ $$C_2 \quad else.$$ Frequentist Decision Boundary

Now the problem is that, graphically speaking, the frequentist approach seems to be more precise, while the bayesian one seems to exaggerate the blue area which clearly cover an area that is dominated by red samples. How will you justify the usage of bayesian approach?

Answer 1

Many issues with your analysis and question:

1: the premise "the Bayesian approach is the most correct in Machine Learning" is flawed.

2: It's not actually Bayesian. Base a decision on Pr(C|x) and not Pr(x|C)

3: The LR statistic is UMP for Gaussian mixtures. We expect frequentist and approximate Bayesian inference to be superior.

4: Your Bayes inference looks wrong. It looks like you set C to a single variable representing the Bernoulli proportion of class membership, presumably a beta distribution, which has simply multiplied the same likelihood ratio threshold by an arbitrary factor and led to a mis-calibrated decision rule. Each observation has a class membership, so that is N bernoulli variables, either conditional (hypergeometric) on the proportion or not (binomial) and a prior on the probabilities of class membership taken from a dirichlet distribution. The dirichlet posterior can be updated using the EM estimated centroids for the Gaussian mixture.