0
$\begingroup$

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html

For me, the different is BayesianRidge keep updating alpha and lambda while Ridge only have certain alpha . But what are alpha and lambda in BayesianRidge ?

Since ridge model is defined as $$ \underset{\beta}{\operatorname{arg\,min}}\; \|y - X\beta\|^2_2 + \lambda \|\beta\|^2_2 $$

I can understand alpha in sklearn.linear_model.Ridge is $\lambda$ here . But sklearn.linear_model.BayesianRidge is vague for me.

$\endgroup$
  • $\begingroup$ They appear to influence your prior distribution. $\endgroup$ – Dave Apr 11 at 3:26
0
$\begingroup$

In ridge regression we have some sort of prior over weights $w \vert \gamma \sim \mathcal{N}(0, \gamma^2 \mathbb{I})$ and the likelihood model $y \vert x, w, \sigma \sim \mathcal{N}(\langle w, x \rangle, \sigma^2)$. If we want to stop conditioning on the variances in the prior/likelihood, we can place a prior over each and marginalize these out during prediction. Concretely, what is usually done is to place an inverse Gamma prior over each with respective hyperparameters (because the inverse Gamma is the conjugate prior for the variance of a Normal distribution with known mean).

$$ \sigma^2 \sim \text{Gamma}(\alpha_1, \alpha_2)$$ $$ \gamma^2 \sim \text{Gamma}(\lambda_1, \lambda_2)$$

The hyperparameters are usually set to flat, uninformative priors over $\sigma^2$ and $\gamma^2$ so you don't get some infinite recursion of hyperparameter priors. Looks like SkLearn will estimate these hyperparameters jointly alongside $w$ during the fitting stage. Then during the predictive step for some new test point $x^*$ you can integrate out the variances like so, to be truly 'Bayesian':

$$ p(y^* \vert \mathcal{D}, x^*) = \int d\sigma^2 d\gamma^2 dw p(y^* \vert x^*, w, \sigma^2) p(w, \sigma^2, \gamma^2 \vert \mathcal{D})$$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.