# Bayesian Inference for More Than Linear Regression

I learned about MCMC and variational inference for Bayesian inference, and I would like to try it out in some regression problem. However, all existing related models I know falls within either of the two cases below:

1. Classification problem, such as topic models, but I am only looking for regression problem.
2. Linear regression problem, which I do not see too much benefit compared with classical frequentist estimation of parameters such as confidence interval.

Hence my problem is: Is there any famous/established model in machine learning field that is a non-linear regression problem?

EDIT: I am asking this question because I learnt about the Bayesian method and I feel there is nothing much more it gives compared with the more efficient MLE, except for an (not so good) estimation of confidence interval. Therefore, I am looking for some tasks that Bayesian inference can provide more unique values or performs better than MLE methods. The classical model that came to my mind is parameter estimation for LDA or HDP used for topic models, and perhaps other generative models. However, all those I know is in the domain of classification or clustering. Is there any such established model in the field of regression?

• There are Bayesian analogues of classical ML methods: e.g., Radford Neal's work on Bayesian neural networks or Bayesian Additive Regression Trees (more or less a Bayesian random forest), as well as Bayesian analogues of "meta-algorithms" like stacking and model averaging (see, e.g., projecteuclid.org/euclid.ba/1516093227). Non-parametric techniques like Dirchlet Processes (and combinations / variants thereof) can be used for clustering. May 18, 2018 at 4:14
• More generally, essentially and problem that can be approached in a non-Bayesian way can be approached in a Bayesian way (modulo computational difficulties) by adding priors and using posterior means rather than maximizing something. Whether this observation is useful or just "frequentist-envy" is somewhat debatable, but IMO the most compelling examples of the value of Bayesian approaches are in situations where novel approaches are developed by taking probabilistic / generative reasoning seriously, giving solutions to problems that would otherwise be difficult to approach non-Bayesianly. May 18, 2018 at 4:25
• You've read about linear regression because it's the simplest model, so it's nice for introductory example. What exactly is that your need?
– Tim
May 18, 2018 at 5:29
• You know already about Bayesian Neural Networks. What else do you need? Gaussian Processes are another example of non-linear regression problems, and they’re closely related to BNNs with one hidden layer, sigmoid activation function and a Gaussian prior on weights. Finally, the Variational Autoencoder is another example, and you know about that too. Mixture Density Networks are a famous example by Bishop, and finally the simplest example is probably Probabilistic PCA (well, Bayesian GLMMs are also quite simple and they have distinct advantages over frequentist GLMM). From your other 1/ May 20, 2018 at 8:34
• 2/ questions you seem to be familiar with most of this stuff, so what else are you looking for? Can you explain yourself better? May 20, 2018 at 8:35

I spent a lot of time thinking about your problem and how to answer it. I can actually point out problems which either only have Bayesian solutions, or for which the Bayesian solution is better in a radical way, but I am concerned it wouldn't be useful to you because I think you are really having a mental fog with the problem. I think it is the framework you are putting the solutions in. I will give you a trivial example later that you can look at, but I don't think it is the issue involved.

As a background note, almost my entire formal training is either Pearson and Neyman's Frequentist methods or Fisher's Likelihood method. I have some formal Bayesian training, but almost as a side note. Because of this, I recognize in your question the same types of problems I faced when I ran into a problem with no good non-Bayesian solution. My foray into Bayesian thought was an accident. Also because of this, I try really hard to get people to use the tool that actually is best for the problem at hand and not the one they know. This means sometimes I talk people out of Bayesian methods or into them. This same thinking also applies to things such as artificial neural networks. They are a tool and sometimes a good tool and sometimes inferior.

I am asking this question because I learnt about the Bayesian method and I feel there is nothing much more it gives compared with the more efficient MLE, except for an (not so good) estimation of confidence interval.

This is a serious misunderstanding. I am going to work from back to front on this one. Bayesian credible intervals are not Pearson and Neyman confidence intervals. They do not have the same interpretation, implied meanings, or use.

A confidence interval is probably a misnomer. It would better to say "the interval that is a result of a confidence procedure." The confidence is in the procedure and not the result. A 95% confidence interval of (a,b) is implied to cover a parameter, $$\theta$$, at least 95% of the times the procedure is performed. This is critically important in several ways.

The first and obvious one is that it says nothing about the true location of $$\theta,$$ except that if you behave as if $$\theta$$ is in the interval, then you are guaranteed you will not be made a fool of at least 95% of the time, but cannot say that there is a 95% chance that the true value is inside the interval $$(a,b).$$ Indeed, you can show cases where there is a 100% chance that it cannot be in the interval. Without using a Bayesian method, you can state nothing about your specific sample. There is either a 100% or a 0% chance it is in the interval from Pearson and Neyman's perspective. You do not know which one it is.

If you want a better understanding of confidence intervals and credible intervals I recommend two reads. The first is here on stack exchange, the second in a journal article.

The first is an excellent presentation by Keith Weinstein on the how the two different types of intervals compare to each other under a simple example and why, in some cases, they do not match at all. You can find it at Confidence versus Credible Intervals and Cookies

The second shows cases where the Frequentist result is incredibly inaccurate. You are interpreting confidence intervals as a measure of precision. There is a separate method for calculating precision. They do not measure precision. Let me give you a simple example. Imagine you had the entire population of one school and you split it 50-50 between two researchers in an entirely random manner. Each researcher estimated the sample mean and variance of some parameter of the children.

Without loss of generality assume $$\sigma^2_1<\sigma^2_2$$. The confidence interval of researcher one will be smaller than researcher two if a textbook standard confidence interval were used. Yet the only difference was the random selection. What about being selected into group one made it a more accurate representation of the mean? It isn't. A smaller interval is not more precise than a wider interval in any intrinsic sense of the idea. The boundaries of a confidence interval are a computation from the random numbers generated by the sample, so the boundaries are random numbers.

Morey, R., Hoekstra, R., Rouder, J., Lee, M., & Wagenmakers, E.-J. (2015). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 1–21

To use an extreme case from the Morey article above, consider a submarine sunk on the bottom of the ocean. It is ten meters long. You get one chance to find the sub's rescue hatch or they die. The submarine produces two bubbles, but they produced anywhere along the length of the submarine with uniform probability and only once. The hatch is in the middle.

If two bubble rise and they are together, then the Bayesian likelihood is 10 meters wide and the 50% interval is the point of the bubbles plus or minus 2.5 meters. The Frequentist interval implied from the uniformly most powerful test has zero width, which implies extreme accuracy precisely when it is most inaccurate. Conversely, when the bubbles are exactly 10 meters apart the Bayesian likelihood is of zero width and there is a 100% chance that the hatch is at the midpoint of the bubbles. However, the Frequentist interval implied by the uniformly most powerful test is at its widest of $$\bar{x}\pm\frac{10}{2}.$$

Conversely, the 95% credible interval is an interval with a 95% chance of containing the parameter. Unfortunately, as you can see in the cookie example above, it may not contain the true value 95% of the times you repeat an experiment. The percentage could be quite small. Indeed, the cookie example above includes a case where the 70% credible interval will, upon repetition, only contain the true answer 20% of the time if there were no Bayesian updating.

The first part of the quote was the MLE was more accurate. This is not correct. The MLE is the most likely point estimate. To use Stigler's quote (see reference below):

At a superficial level, the idea of maximum likelihood must be prehistoric: early hunters and gatherers may not have used the words method of maximum likelihood'' to describe their choice of where and how to hunt and gather, but it is hard to believe they would have been surprised if their method had been described in those terms. It seems a simple, even unassailable idea: Who would rise to argue in favor of a method of minimum likelihood, or even mediocre likelihood?

Stigler, Stephen M. The Epic Story of Maximum Likelihood. Statist. Sci. 22 (2007), no. 4, 598-620. doi:10.1214/07STS249. https://projecteuclid.org/euclid.ss/1207580174

Being the most likely solution is not the same thing as being the most accurate solution. Stein's lemma shows that this is not the case. See:

Stein, C. (1956). "Inadmissibility of the usual estimator for the mean of a multivariate distribution". Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability. 1. pp. 197–206.

Indeed, if accuracy is your goal, then as long as you use your best estimate of the prior density for the parameters, all Bayesian estimators are guaranteed to be admissible. Non-Bayesian estimators are only admissible if they either map to the Bayesian estimator in every sampling or converge at the limit. If accuracy is your goal, then Bayesian estimators cannot be stochastically dominated by non-Bayesian estimators. Non-Bayesian estimators can be stochastically dominated by Bayesian estimators.

However, the goal may not be accuracy. It may be to assure an absence of bias. If the goal is not accuracy, but protection from bias, a guaranteed level of protection against false positives and an ability to control the level of false negatives, then you should use Pearson and Neyman's minimum variance unbiased estimator. Generally, the MLE is not unbiased and with a proper prior the Bayesian estimator is nearly guaranteed to be biased.

What does the MLE provide then if not unbiasedness or accuracy, it provides a solution to the epistemological question of whether a null hypothesis is likely to be false by providing a measure of the "weight of the evidence against" the null. That weight is called the p-value. Pearson and Neyman use an acceptance region or a rejection region based on a cutoff value $$\alpha$$ and so the p-value contains no additional information in that case. In the Likelihoodist methodology, however, you are looking at the strength of the evidence with the p-value.

Now let me give you a simple case that I have provided more fully elsewhere on stack exchange, but I cannot find now. Nonetheless, I will let you solve it computationally and you will find that there may be serious unexpected consequences by your choice of method.

Imagine that the Congress authorized the creation of biased and unbiased US quarters. The bias for heads is either $$\{1/3,1/2,2/3\}.$$ The probability of being randomly chosen from the minted coins is $$\{25\%,50\%,25\%\}$$ for each respective choice. You can either construct a solution where you are ignorant of the prior distribution or you know it.

Now as a short thought experiment you were a bookie. You can either choose the MLE or the Bayesian solution. The Bayesian solution is a discrete choice among the three possible parameters, but the MLE is just $$\frac{k}{n}$$ or the number of heads divided by the number of trials.

Let us imagine you tossed the coin 8 times to gather information on the true value of the parameter. You saw the results as well as the gamblers. Based only on this sample, what odds would you post for each possible outcome for a future set of 8 trials? What if you saw six heads or zero heads?

For zero heads, the MLE is for a double-tailed coin with no variance at all. The MAP estimator is for a $$1/3$$ bias under a uniform prior. If you use the Bayesian and Frequentist predictive distributions rather than plug in the MLE or MAP estimator you will get a big increase in Bayesian accuracy, but except in the corner cases no increase in the Frequentist accuracy. Indeed, the Frequentist odds are so misstated that you can guarantee that you can place arbitrage bets against the bookie and never lose.

Choose Bayesian methods when accuracy is your primary concern or when gambling is involved. Use MLE when you need to reject a false understanding of the world. Use Pearson and Neyman methods when you need a clean estimate that provides you some guaranteed level of protection from making the wrong decisions while controlling for enough power to be able to say you are making a good decision.

To give you a regression example that I have run. Go to the FDIC database of bank financial statements from before the crisis. Look at the impact of Federal Reserve tightening or loosening on risk-taking behavior as a function of leverage. You will get very different parameter results due to the fact that some banks fail, some are started, and some just have missing information at times. Because of how Bayesian updating works, you will end up with different parameter estimates in the sense that both the values and some of the signs are different, and the accuracy is different.

Although that is an example, if I were just learning, I wouldn't try that problem. Build a toy problem, as above, with the coins. The simplest solution to get large differences is to build a boundary condition in. Most non-Bayesian regression methods presume the entire real number line is a possible value, but if there are known restrictions they will include impossible answers. It can be the case, sometimes, that you can build restrictions into a regression by limiting support, but not always and not without lots of headaches. Bayesian methods just assign a 0% chance in impossible regions.