0
$\begingroup$

I am trying to get a better understanding of the math behind logistic regression.

Logistic regression is looking to give a prediction based on data likelihood. From my understanding it is making a posterior prediction.

Suppose we have a two-class model. So this becomes the equation

\begin{align} p(C_1|x) &= \frac{p(x|C_1)p(C_1)}{p(x|C_1)p(C_1)+p(x|C_2)p(C_2)}\\ &=\frac{1}{1+\exp(-a)} = \sigma(a) \end{align}

where $a= \ln \frac{p(x|C_1)p(C_1)}{p(x|C_2)p(C_2)}$

My questions are what does $a= \ln \frac{p(x|C_1)p(C_1)}{p(x|C_2)p(C_2)}$ intuitively mean? Does it imply that the $\theta x$ of our model, is trying to estimate the ratio in $a$? And why in the first step is the equation equivalent to the same as the naive bayes classifier; how does logistic regressor distinguish itself from the naive bayes if the background math is the same

Improve this question
$\endgroup$
1
  • $\begingroup$ Try substituting $a$ in the expression for $\sigma(a)$. You will see it's just definitional given the first line, there isn't any intuition in it except as a transform of the first expression. The logistic regressor is equivalent to this by definition; we could use other expressions (see probit regression) for an expression. It gets its name from the Logistic distribution, the standardized form of which has cumulative density function equal to $1/(1+\exp(-x))$. $\endgroup$
    – jbowman
    Jan 22, 2018 at 4:02

1 Answer 1

Reset to default
0
$\begingroup$

Basic workflow of logistic regression is:

  1. Calculate log-odds of success (success means that your observation belongs to the target class of interest)
  2. Convert log odds to percentage likelihood
  3. Apply some threshold to your percentage likelihood in order to apply a predicted class label of "success" or "failure"

Your "a" term are known as the log-odds. These are the log of the odds that a sample belongs to a given class given that it possesses a set of predictors, to the odds that it does not belong to a certain class given its predictors. Think of it as the raw output from a logistic regression method in your native statistical software package (although sometimes you could also be given a probability, or even the class label directly, the log odds are driving all of these secondary outputs).

We cannot directly apply a linear regression to a classification problem since it violates the assumption that the response variables be continuous (i.e. exists between - infinity and + infinity). What we can say is that the odds in favor of an event occurring (in this case that your observation belongs to a class of interest), are positive and (usually) non-zero numbers. This means you can also calculate the log of those odds (the log odds from earlier). The beauty of the log odds is that they are continuous values that we can apply linear regression to.

The remainder of the work comes from finding the weights of your coefficients that maximize the probability of an observation belonging to the success class if it is indeed a success, and minimize the probability that are assigned to non-successes. This is controlled through the coefficient weights.

If you exponentiate the log-odds (your 'a' term) it is straightforward to calculate the probability. 

           Probability = (odds)/(odds+1)

Given that we have log-odds:

           Probability = exp(a)/(exp(a)+1)

Divide out the exp(a) term and the probability statement reduces to your 

            sigma(a) = 1/ 1+exp(-a)
Improve this answer
$\endgroup$
5
  • $\begingroup$ Hi thanks for your answer. So basically the log odds is the unoptimized version of the logistic regression? $\endgroup$
    – mayankmehtani
    Jan 22, 2018 at 5:44
  • $\begingroup$ It's not that log-odds are "unoptimized", they just need to be converted to a probability and then to a class prediction. $\endgroup$
    – rmrouse88
    Jan 22, 2018 at 6:18
  • $\begingroup$ makes sense, but what is the relation of the log odds to our weights? $\endgroup$
    – mayankmehtani
    Jan 22, 2018 at 6:33
  • $\begingroup$ Your initial intuition was close to correct, except the theta*X terms are not just estimating the ratio: They are estimating the log of that ratio. The thetas are the weights, and the choice of weights will generate different values of log-odds for each observation depending on the value of the predictors associated with an observation. Larger value of log odds translates to higher probability of target class membership. $\endgroup$
    – rmrouse88
    Jan 22, 2018 at 13:51
  • $\begingroup$ Hi thanks for the reply. We could write the same equation on the posterior probability using the naive bayes classifier, correct? Does estimating the log likelihoood with an optimized theta, using gradient descent, differentiate this from the naive bayes model as this will not double count correlated features.. $\endgroup$
    – mayankmehtani
    Jan 22, 2018 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.