Can someone explain to me the Bayesian classification model?

Question

I often read about converting from a normal classification model like logistic regression and then using an equivalent Bayesian model. As I understood, it's somehow the same model but with a different motivation somehow. I read about defining the model of $P(y|x,\theta)$ using a Gaussian and also using a Gaussian prior to define $\theta$, but I can't understand the idea of that, or why it's valid. Why not other distribution?!

Can someone please explain the model to me? if not, at least recommend some tutorials or not too complicated papers that explain the idea and the motivations and proofs behind the model.

Answer 1

The Bayesian approach proposes using our previous expereince or knoweldge, if available. For example, If I ask you what is the probability of having a head in a coin flip, you will probably say 50%. However, if you actually flip the coin 3 times, you might get 1/3, 2/3 or even 3/3 heads.

From a frequentists point of view (opposed to Bayesians), the probabilities would be (1/3 ~= 33%, 66%, 100%). The Baysians, on the other hand, say that we already previously know that the probability of having a head in a coin flip is 50% (our prior knoweldge). Let's use it! Now, the probabilities can be corrected by this prior factor so that, the probability of having 1/3 heads = (1/3)*(1/2) = 16% (and not 33% as we got before).

From a first glance, the Baysian approach seems very attractive as it allows us to make use of our prior knowedlge. However, if we are not able to quatify our prior knoweldge with good estimates, then, we might deviate from the actual description of our surrounding environment.

A small comment for the Gaussain distribution choice, is that it simplifies learning the parameters of the model. In particualr, it can be shown that if the distribution of the prior knowedlge is Gaussian, then the posterior distribution is also a Gaussain (conjugate prior). Needless to say, this might not be the best justification to use a Gaussian model as the assumptions of the Gaussian distribution might not hold for a particular dataset, yet, it performs well in many cases.