1
$\begingroup$

I'm trying to read this article about Bayesian logistic regression.

In general, to classify instances, they use: $p(y=+1 |\beta) = \sigma(\beta^TX) $ (where $\sigma$ is obviously the sigmoid function). Now they need to find the values of $\beta$.

They then say that to avoid overfitting, they'll use a prior distribution on $\beta$ specifying that each value of $\beta$ is likely to be near 0. They then impose a univariate Gaussian prior with mean =0 and variance >0 on each parameter $\beta$. (Section 3).

What I don't understand is:

  1. How does specifying $\beta$ to be near 0 help avoid overfitting?

  2. How could one assume a Gaussian distribution on a set of parameters that they're supposed to estimate?

Any kind of help would be greatly appreciated...

$\endgroup$

1 Answer 1

4
$\begingroup$

1) The overfitting will come if a huge number of parameters $\beta$ is significantly different from zero. Putting a prior distribution on each $\beta$ centered on zero is, in some sense, as forcing all the parameters to be non significantly and then only the ones that are strongly significant can have a posterior distribution non close to the zero.

2) If you are working in a bayesian framework, you have to put a prior distribution on each parameter you want to estimate. The prior can be any distribution (uniform, beta, gamma), depends on the type of variable you are putting the prior distribution on. Using a normal prior centered around 0 is like said that departure from zero in negative and positive value are equally likely.

$\endgroup$
1
  • $\begingroup$ (+1) This is a solid answer. I would suggest making mention of the key term "shrinkage" and drawing connections to regularization in the classical/MLE context: that is, this estimation procedure allows some estimates to be precisely zero, rather than ML fits which might be noisily estimated somewhere around zero. $\endgroup$
    – Sycorax
    Jul 25, 2014 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.