1
$\begingroup$

We know that marginal likelihood has the following form in Bayesian linear regression, $$ \mathbf{K} = \sigma_w^2XX^T + \sigma_n^2I\\ p(\mathbf{y}|X) \sim \mathcal{N}(0, K)\\ \log p(\mathbf{y}|X) = -\frac{1}{2} \mathbf{y}^{T} K^{-1} \mathbf{y} - \frac{1}{2}\log|K| - \frac{n}{2}\log (2 \pi) $$

Can we estimate $\sigma_n$ and $\sigma_w$ in closed form by maximizing log marginal likelihood or with any other way (other than numerical methods such as gradient descent)? I am also interested to know if this is possible for Gaussian process regression where $K$ could be RBF or Matern or any other kernel?

$\endgroup$

2 Answers 2

1
$\begingroup$

That is the exact procedure used in GP. Kernel parameters obtained by maximizing log marginal likelihood. You can use any numerical opt. method you want to obtain kernel parameters, they all have their advantages and disadvantages. I dont think there is closed form solution for parameters though.

$\endgroup$
1
  • $\begingroup$ I see. Thanks for your answer. I could relate it to GP that way but intuitively thought that linear regression being the simplest model may have some trick. $\endgroup$ Mar 30, 2021 at 15:42
0
$\begingroup$

This blog post gives the answer I was looking for. There is a method to estimate $\sigma_w$ and $\sigma_n$ parameters for linear regression that is computationally far better than fitting a GP with linear kernel.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.