7
$\begingroup$

I'm a bit confused whether these two are the same concept. If they are different what's the difference?

Thanks!

$\endgroup$
  • $\begingroup$ @Clarinetist You should post your comments as an answer. $\endgroup$ – Patrick Coulombe Jul 12 '15 at 22:48
  • $\begingroup$ @PatrickCoulombe Done. $\endgroup$ – Clarinetist Jul 12 '15 at 23:13
5
$\begingroup$

The other answers are good. However, to clarify the intuition as well as give some further details:

  • In logistic regression, you maximize the likelihood function $p(y|\beta_{0},\beta_{1},x)$ (find MLE). That is, you find the weights $\beta_{0},\beta_{1}$ that maximizes how likely your observed data is. There is no closed form solution to the MLE, so you need to use iterative methods. This gives you a single point estimate of our weights.
  • In bayesian logistic regression, you start with an initial belief about the distribution of $p(\beta_{0},\beta_{1})$. Then $p(\beta_{0},\beta_{1}|x,y)\propto p(y|\beta_{0},\beta_{1},x)p(\beta_{0},\beta_{1})$. That is, the posterior, which is our updated belief about the weights given evidence, is proportional to our prior (initial belief) times the likelihood. We can't evaluate the closed form posterior, but can approximate it by sampling or variational methods. This gives us a distribution over the weights. For instance, if we use a normal approximation for both $\beta_{0}$ and $\beta_{1}$ using variational methods, then we'll get a mean and variance for $\beta_{0}$, and one for $\beta_{1}$ as well.

For further detail on both techniques, these scribe notes of a lecture are excellent http://web.cse.ohio-state.edu/~kulis/teaching/788_sp12/scribe_notes/lecture6.pdf.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Maximum likelihood estimation does provide a point estimate of the parameters, but one can also and should provide an estimate of uncertainty by using normal-approximation justified by the large sample properties of maximum likelihood estimators. Bayesian logistics regressions starts with prior information not belief. If you have no prior information you should use a non-informative prior. Gelman et al. recommend default logistic regression Cauchy priors with scale = 0.1 for intercept terms and scale = 0.4 for slope terms. $\endgroup$ – Dalton Hance Jul 13 '15 at 20:16
  • $\begingroup$ Thank you. Can you clarify the meaning of prior information? $\endgroup$ – mlstudent Jul 13 '15 at 21:10
  • $\begingroup$ It's a matter of semantics mostly. Prior belief and prior information are two different English language phrases for the same concept: the probability distribution of the parameters you take with you into the model. I emphasize the term information over belief because you really should have some justification for it (existing literature, expert opinion, a pilot study, or even an empirical estimate) other than your own faith. $\endgroup$ – Dalton Hance Jul 13 '15 at 22:12
  • 2
    $\begingroup$ If the link doesn't work: web.archive.org/web/20150409022746/http://… $\endgroup$ – 190290000 Ruble Man Jul 6 '17 at 13:25
5
$\begingroup$

Suppose you have a set of binary observations $Y_i$ for $i=1,\ldots,n$ and, for each observation, an associated explanatory variable $X_i$. Logistic regression assumes $$ Y_i \stackrel{ind}{\sim} Ber(\pi_i), \quad \ln\left(\frac{\pi_i}{1-\pi_i}\right)=\beta_0+\beta_1 X_i.$$ If you are obtaining point estimates of the parameters via maximum likelihood, then you just use the assumptions above. But, if you are obtaining estimates of the parameters using a Bayesian approach, then you need to define a prior for $\beta_0$ and $\beta_1$, call it $p(\beta_0,\beta_1)$. This prior along with the logistic regression assumptions above is Bayesian logistic regression.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I don't claim to be an expert on Logistic Regression. But I imagine it goes something like this - suppose $Y$ is a binary random variable taking on either the value $0$ or $1$. Define $$\pi=\mathbb{P}\left(Y=0∣X\right)\text{,}$$ where $X$ is the independent variable (I'm assuming only one predictor for simplicity). Then logistic regression assumes the form $$\ln\left(\dfrac{\pi}{1-\pi}\right)=\beta_0+\beta_1 X + \epsilon$$ where $\epsilon$ is independent from $X$ and has mean $0$, and the $\beta_i$ are estimated using maximum likelihood. With Bayesian logistic regression, I imagine you use something like $$\pi = \dfrac{\mathbb{P}\left(X = x\mid Y = 0\right)\mathbb{P}\left(Y = 0\right)}{\displaystyle\sum\limits_{j}\mathbb{P}\left(X = x\mid Y = j\right)\mathbb{P}\left(Y = j\right)}$$and assign something for the distribution of $X \mid Y = j$ and a prior distribution for $Y$. This is, from my limited understanding, I believe the basis of Linear Discriminant Analysis.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.