How is BayesianRidge different from Ridge?

Question

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html

For me, the different is BayesianRidge keep updating alpha and lambda while Ridge only have certain alpha . But what are alpha and lambda in BayesianRidge ?

Since ridge model is defined as $$ \underset{\beta}{\operatorname{arg\,min}}\; \|y - X\beta\|^2_2 + \lambda \|\beta\|^2_2 $$

I can understand alpha in sklearn.linear_model.Ridge is $\lambda$ here . But sklearn.linear_model.BayesianRidge is vague for me.

Answer 1

In ridge regression we have some sort of prior over weights $w \vert \gamma \sim \mathcal{N}(0, \gamma^2 \mathbb{I})$ and the likelihood model $y \vert x, w, \sigma \sim \mathcal{N}(\langle w, x \rangle, \sigma^2)$. If we want to stop conditioning on the variances in the prior/likelihood, we can place a prior over each and marginalize these out during prediction. Concretely, what is usually done is to place an inverse Gamma prior over each with respective hyperparameters (because the inverse Gamma is the conjugate prior for the variance of a Normal distribution with known mean).

$$ \sigma^2 \sim \text{Gamma}(\alpha_1, \alpha_2)$$ $$ \gamma^2 \sim \text{Gamma}(\lambda_1, \lambda_2)$$

The hyperparameters are usually set to flat, uninformative priors over $\sigma^2$ and $\gamma^2$ so you don't get some infinite recursion of hyperparameter priors. Looks like SkLearn will estimate these hyperparameters jointly alongside $w$ during the fitting stage. Then during the predictive step for some new test point $x^*$ you can integrate out the variances like so, to be truly 'Bayesian':

$$ p(y^* \vert \mathcal{D}, x^*) = \int d\sigma^2 d\gamma^2 dw p(y^* \vert x^*, w, \sigma^2) p(w, \sigma^2, \gamma^2 \vert \mathcal{D})$$.