Let us first focus on classification. We want to build a function that can tell us the class of any new instance we feed it. In most cases, it will not be unambiguous from the training data which class we should assign, so it sometimes makes sense to output a probability distribution over classes, rather than just the best guess. Of course, given such a distribution we can specialise to the most likely class simply by selecting the class with the highest probability.
Often we want to restrict the possible functions we consider some way. For instance, we may have a set of parameters $w$ that we can tune. Each specific choice of parameters could correspond to a specific classification function, that assigns a specific class to each possible input.
Bayes' theorem gives us a way to compute probability distributions over such functions. It turns your prior beliefs about what the parameters of your function could be, as represented through $p(w)$, and the probabilities of the observed data under these beliefs, $p(D|w)$, into the probabilities of the parameters under the data, $p(w|D)$. This last term is a probability of parameters of your function, which immediately implies a probability distribution over functions. This means that for each input, we have a probability distribution over outputs.
Regression is similar to classification, but there the outputs are not integers (classes) but real-valued numbers or vectors. Again, we can obtain probability distributions over parameters of such real-valued functions, which imply probability distributions over the outputs for any given input.
Sometimes, you may not need full probability distributions for each input. If you only need a best-guess, other (machine-learning) methods exist that do not build intermediate probability distributions explicitly. So, although Bayes' theorem can be used for such problems, it is not the only possible solution. You already mention an alternative: k-nearest neighbours gives you a way to output a best-guess answer, but it does not (explicitly) build probability distributions.