Estimation of Bayesian Ridge Regression

Question

According to scikit-learn, by using a probabilistic model :

$p(y|X,\omega,\alpha) = \mathcal{N}(y|X\omega,\alpha)$

with $\omega$ given by a spherical Gaussian: $p(\omega|\lambda) = \mathcal{N}(\omega|0,\lambda^{-1}\mathbf{I_p})$

it is now a Bayesian model of ridge regression. So can i say that the estimation of this model on unknown data $X^*$ is a probability distribution on y with mean $\mu$ = $X\omega$ and variance $\sigma^2 = \alpha$ or $\sigma^2=\lambda^{-1}\mathbf{I_p}$ ? What exactly do $\alpha$ and $\lambda$ do in the equations ?

Answer 1

What the description in the sklearn documentation says is that the model is a regression model with extra regularization parameter for the coefficients. The model is

$$\begin{align} y &\sim \mathcal{N}(\mu, \alpha^{-1}) \\ \mu &= X\omega \\ \omega &\sim \mathcal{N}(0, \lambda^{-1}\mathbf{I}_p) \\ \alpha &\sim \mathcal{G}(\alpha_1, \alpha_2) \\ \lambda &\sim \mathcal{G}(\lambda_1, \lambda_2) \end{align}$$

So $y$ follows normal distribution (the likelihood function) parametrized by mean $\mu = X\omega$ and variance $\alpha^{-1}$. Where we choose Gamma priors for $\alpha$ and regularizing parameter $\lambda$, the distributions have hyperpriors $\alpha_1, \alpha_2, \lambda_1, \lambda_2$. The regression parameters $\omega$ have independent Gaussian priors with mean $0$ and variance $\lambda^{-1}$, so $\lambda$ serves as a regularization parameter (it is a precision parameter, so the larger $\lambda$, the $\omega$ values are a priori assumed to be more concentrated around zero).

Answer 2

I have not looked at the scikit learn documentation closely yet, but, from what I know about bayesian ridge, the learned alpha and lambda values (using scikit learned terminology for these) are hyperparameters of the bayesian ridge model. Alpha corresponds to the noise in your estimate of the target and lambda is the estimated precision of the weights. The equations above are good descriptions of these.

These are learned in scikit learn by assigning a gamma priors to the hyperparameters and marginalizing over these hyperparameters. Then, because the resultant integral is analytically intractable, we need to use some approximation to find it (but I am not sure what they use... maybe Laplace approximation). After this, we have effectively solved for the hyperparameters without splitting the dataset and using a grid search method!

We solve for these hyperparameters because they set our quadratic regularization term in the Bayesian ridge regression. Following Bishop's Pattern Recognition and Machine Learning, the posterior distribution w.r.t w (the weight vector) shown in equation (eq. 3.55):

We can see that this look similar to the minimization of the sum-of-squares error function with a quadratic regularization term that is equivalent to alpha/beta or, in scikit learn's terminology, lambda/alpha. See below the minimization of the sum-of-squares error function with a quadratic regularization term (eq. 3.27) for comparison. Where Bishops alpha = scikit learn's lambda and Bishops beta = scikit learn's alpha:

Another good source for understanding bayesian linear regression is: http://krasserm.github.io/2019/02/23/bayesian-linear-regression/