I'm a bit confused whether these two are the same concept. If they are different what's the difference?



3 Answers 3


The other answers are good. However, to clarify the intuition as well as give some further details:

  • In logistic regression, you maximize the likelihood function $p(y|\beta_{0},\beta_{1},x)$ (find MLE). That is, you find the weights $\beta_{0},\beta_{1}$ that maximizes how likely your observed data is. There is no closed form solution to the MLE, so you need to use iterative methods. This gives you a single point estimate of our weights.
  • In bayesian logistic regression, you start with an initial belief about the distribution of $p(\beta_{0},\beta_{1})$. Then $p(\beta_{0},\beta_{1}|x,y)\propto p(y|\beta_{0},\beta_{1},x)p(\beta_{0},\beta_{1})$. That is, the posterior, which is our updated belief about the weights given evidence, is proportional to our prior (initial belief) times the likelihood. We can't evaluate the closed form posterior, but can approximate it by sampling or variational methods. This gives us a distribution over the weights. For instance, if we use a normal approximation for both $\beta_{0}$ and $\beta_{1}$ using variational methods, then we'll get a mean and variance for $\beta_{0}$, and one for $\beta_{1}$ as well.

For further detail on both techniques, these scribe notes of a lecture are excellent http://web.cse.ohio-state.edu/~kulis/teaching/788_sp12/scribe_notes/lecture6.pdf.

  • 1
    $\begingroup$ Maximum likelihood estimation does provide a point estimate of the parameters, but one can also and should provide an estimate of uncertainty by using normal-approximation justified by the large sample properties of maximum likelihood estimators. Bayesian logistics regressions starts with prior information not belief. If you have no prior information you should use a non-informative prior. Gelman et al. recommend default logistic regression Cauchy priors with scale = 0.1 for intercept terms and scale = 0.4 for slope terms. $\endgroup$ Jul 13, 2015 at 20:16
  • $\begingroup$ Thank you. Can you clarify the meaning of prior information? $\endgroup$
    – mlstudent
    Jul 13, 2015 at 21:10
  • $\begingroup$ It's a matter of semantics mostly. Prior belief and prior information are two different English language phrases for the same concept: the probability distribution of the parameters you take with you into the model. I emphasize the term information over belief because you really should have some justification for it (existing literature, expert opinion, a pilot study, or even an empirical estimate) other than your own faith. $\endgroup$ Jul 13, 2015 at 22:12
  • 3
    $\begingroup$ If the link doesn't work: web.archive.org/web/20150409022746/http://… $\endgroup$ Jul 6, 2017 at 13:25
  • $\begingroup$ Are there any specific cases when we should use Bayesian LR over LR? $\endgroup$ Jan 17, 2022 at 22:23

Suppose you have a set of binary observations $Y_i$ for $i=1,\ldots,n$ and, for each observation, an associated explanatory variable $X_i$. Logistic regression assumes $$ Y_i \stackrel{ind}{\sim} Ber(\pi_i), \quad \ln\left(\frac{\pi_i}{1-\pi_i}\right)=\beta_0+\beta_1 X_i.$$ If you are obtaining point estimates of the parameters via maximum likelihood, then you just use the assumptions above. But, if you are obtaining estimates of the parameters using a Bayesian approach, then you need to define a prior for $\beta_0$ and $\beta_1$, call it $p(\beta_0,\beta_1)$. This prior along with the logistic regression assumptions above is Bayesian logistic regression.


I don't claim to be an expert on Logistic Regression. But I imagine it goes something like this - suppose $Y$ is a binary random variable taking on either the value $0$ or $1$. Define $$\pi=\mathbb{P}\left(Y=0∣X\right)\text{,}$$ where $X$ is the independent variable (I'm assuming only one predictor for simplicity). Then logistic regression assumes the form $$\ln\left(\dfrac{\pi}{1-\pi}\right)=\beta_0+\beta_1 X + \epsilon$$ where $\epsilon$ is independent from $X$ and has mean $0$, and the $\beta_i$ are estimated using maximum likelihood. With Bayesian logistic regression, I imagine you use something like $$\pi = \dfrac{\mathbb{P}\left(X = x\mid Y = 0\right)\mathbb{P}\left(Y = 0\right)}{\displaystyle\sum\limits_{j}\mathbb{P}\left(X = x\mid Y = j\right)\mathbb{P}\left(Y = j\right)}$$and assign something for the distribution of $X \mid Y = j$ and a prior distribution for $Y$. This is, from my limited understanding, I believe the basis of Linear Discriminant Analysis.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.