Confusion when Learning Parameters in BAYESIAN MODELS

Question

I'm learning Bayesian Models but i still have some issues with the training of the parameters. These are my two questions :

1) Recall the Bayesian formula : $$p(\theta|X) = \frac{ p(X|\theta) \; p(\theta) }{p(X)}$$ It seems that we learn the hyper parameters of our model by maximizing the likelihood. In the case of a Gaussian Process for example, we modify the parameter of the kernel by maximizing the likelihood which is differentiable with respect to theta, so we can use gradient based methods... (see 5.4.1 in the Gaussian Process Book)

But my question is : if we maximize only the likelihood, we are back to a frequentist approach no ? i thought that the whole thing with Bayesian Methods was to consider both the likelihood and the prior ... why not choosing the parameters by maximizing the Maximum a Posteriori ?!

---

2) How do we modify the parameters of a Bayesian Model if the likelihood is not differentiable with respect to the parameters. In other words, when we use an MCMC to approximate the posterior distribution, we have no way to get the derivative of the likelihood because we have only samples that follow the target distribution... In this case how do we learn the optimal parameters ?

In other words, how is it possible to get the derivative of the likelihood (or the MAP depending the answer of the first question) with respect to some parameters if we only have samples that follow the posterior distribution ?

In a bayesian neural network for example, we have to modify the mean and the standard deviation of all the weight distributions. This paper ( Weight Uncertainty in Neural Networks) presents a method with an approximation of the posterior distribution with a variationnal family. (Bayes by Backprop) In this case, indeed, we have an expression of the variationnal free energy that we can differentiate with respect to some parameters. But in the case of a Monte Carlo, how do we modify the weights parameters ?

Thank you so much for your help !

Answer 1

But my question is : if we maximize only the likelihood, we are back to a frequentist approach no ? i thought that the whole thing with Bayesian Methods was to consider both the likelihood and the prior

You are correct, if we only maximize likelihood, then it's maximum likelihood, not Bayesian approach. In Bayesian setting we maximize posterior (i.e. prior times the likelihood), or even estimate the distribution rather then finding point estimate. On another hand, if you choose uniform prior $p(\theta) \propto 1$, then $p(X|\theta) \times 1$ is the same as maximizing likelihood alone. Notice that in section 5.4.1 they say

In this section we first apply the general Bayesian inference principles from section 5.2 to the specific Gaussian process model, in the simplified form where hy- perparameters are optimized over. We derive the expressions for the marginal likelihood and interpret these.

So they describe maximizing the likelihood alone as a simplified example.

As about your second question, I must say that I don't fully understand what you mean. We have many MCMC algorithms that let us sample form posterior distribution given that we specify the model in terms of likelihood and priors, only some of them need you to take the derivatives.

But in the case of a Monte Carlo, how to we modify the weights parameters ?

We don't modify anything. Likelihood and prior are enough to specify the posterior distribution (recall Bayes theorem). Given them, we use MCMC to take samples from the posterior distribution. Those samples enable us to characterize the posterior distribution even if it is not analytically tracable (otherwise we wouldn't need the samples, we would just derive the posterior analytically).

Even if you are interested only in the mode of the posterior distribution, i.e. maximum a posteriori estimate, and if you want to optimize such functions that are not differentiable, there are many optimization algorithms that don't need derivatives.