How does the Bayes' theorem equation generalize all sorts of regression/classification models?

I have been reading “Pattern Recognition & Machine Learning” written by Christopher M. Bishop for some time, but I am still a beginner. I wish to get a bigger view that summarizes regression and classification models.

The book begins with Bayes’ theorem stating that:

$$p(\textbf{w} \mid D) = \frac{p(D \mid \textbf{w})p(\textbf{w})}{p(D)}$$ Where $D$ is the observed data, $w$ is model parameters.

From this equation, how does it generalize all sorts of regression or classification models? Specifically, I wish to know how it relates to following model types:

• Clustering, e.g. k-means clustering
• Density estimation, e.g. k-nearest neighbours
• Classification, e.g. Gaussian classifier
• Regression, e.g. Bayesian linear regression

I know each of the technique, but how does the likelihood and prior probability functions in the Bayesian relate to them?

• The main thing it will do is provide measures of accuracy, that (in principle) take account of all sources of error. ML typically just gives you a point estimate, but no measure of how accurate you expect that particular estimate to be. – probabilityislogic Apr 8 '14 at 10:13

1 Answer

Let us first focus on classification. We want to build a function that can tell us the class of any new instance we feed it. In most cases, it will not be unambiguous from the training data which class we should assign, so it sometimes makes sense to output a probability distribution over classes, rather than just the best guess. Of course, given such a distribution we can specialise to the most likely class simply by selecting the class with the highest probability.

Often we want to restrict the possible functions we consider some way. For instance, we may have a set of parameters $w$ that we can tune. Each specific choice of parameters could correspond to a specific classification function, that assigns a specific class to each possible input.

Bayes' theorem gives us a way to compute probability distributions over such functions. It turns your prior beliefs about what the parameters of your function could be, as represented through $p(w)$, and the probabilities of the observed data under these beliefs, $p(D|w)$, into the probabilities of the parameters under the data, $p(w|D)$. This last term is a probability of parameters of your function, which immediately implies a probability distribution over functions. This means that for each input, we have a probability distribution over outputs.

Regression is similar to classification, but there the outputs are not integers (classes) but real-valued numbers or vectors. Again, we can obtain probability distributions over parameters of such real-valued functions, which imply probability distributions over the outputs for any given input.

Sometimes, you may not need full probability distributions for each input. If you only need a best-guess, other (machine-learning) methods exist that do not build intermediate probability distributions explicitly. So, although Bayes' theorem can be used for such problems, it is not the only possible solution. You already mention an alternative: k-nearest neighbours gives you a way to output a best-guess answer, but it does not (explicitly) build probability distributions.