What is Significant about the Bayesian Approach in Machine Learning?

Question

In machine learning, I've heard a lot of references to the "Bayesian Approach" or a "Bayesian Model." I found this question and it seems to suggest that a Bayesian Model, very broadly speaking, attempts to estimate a posterior probability distribution from a data set, given a prior. But isn't that what machine learning is in general, trying to estimate some distribution function given data? Maybe I'm just not understanding what a Bayesian model really is, but, doesn't that imply that all models that return a probability distribution (i.e. your vanilla Logistic Regression model) would fall under this "Bayesian" category? I somehow don't think that's true.

In any case then, do Bayesian Models offer some sort of inherent advantages or disadvantages over non-Bayesian models? I realize that's a very much dependent on the context of the question, but I'm asking if the model's Bayesian quality inherently imparts different properties on the model, regardless of the specifics of the model outside of it being Bayesian?

Answer 1

In any case then, do Bayesian Models offer some sort of inherent advantages or disadvantages over non-Bayesian models?

Simple answer: no. It's just a different type of thinking. Bayesian modeling just gives you a posterior density for the parameters your estimating. Usually, frequentist models like generalized linear models are simpler to compute but do not provide a posterior distribution because frequentists do not believe the parameters are random, but rather fixed things you have to estimate. Generally, if the sample size is large enough (about 300+ rows of data from my experience), both Bayesian models and frequentist models come out with the same parameter estimates.

In simple terms, a frequentist views the data as the random variable, where as the Bayesian views the parameters you're estimating as the random variable.

You should review some fundamental ideas about Bayesian statistical analyses before trying to apply them. A good reference that I recommend is Bayesian Ideas and Data Analysis: An Introduction for Scientists and Statisticians. It's a fairly easy read, with math at the Master's graduate level.

Answer 2

Agree with the previous answer. In fact, the idea behind almost any Bayesian model is having random variable, which can be distributed according to prior and posterior distributions. It doesn't imply that Bayesian approach works better or worse than non-Bayesian one, but usually it requires more data (for prior distribution of parameters) and in many cases (not all of them) it needs the variables we're working with to be independent.

Moreover, Bayesian approaches can be implemented only for parametric models; they're easier to interpret than other types of ML techniques, but the choice of probability distributions and their parameters is crucial here. My opinion is subjective one, but in practice non-Bayesian approaches are used more frequently in real-life applications, where we have no idea about the source of data coming from and have a really bad understanding of probability distributions suitable for the data we have.

Overall, for almost any of the Bayesian techniques there's an alternative one. It is more a matter of choice and situation, than a matter of a result, as my experience tells me.