24
$\begingroup$

Recently I began studying machine learning, however I failed to grasp the intuition behind logistic regression.

The following are the facts about logistic regression that I understand.

  1. As the basis for hypothesis we use sigmoid function. I do understand why it's a correct choice, however why it's the only choice I don't understand. Hypothesis represents the probability that the appropriate output is $1$, therefore the domain of our function should be $[0,1]$, this is the only property of sigmoid function I found useful and appropriate here, however many functions satisfy this property. In addition, sigmoid function has a derivative in this form $f(x)(1-f(x))$, but I don't see the utility of this special form in logistic regression.

    Question: what so special about sigmoid function, and why we cannot use any other function with domain $[0,1]$?

  2. The cost function consists of two parameters ${\rm Cost}(h_{\theta}(x),y)=-\log(h_{\theta}(x))$ if $y=1, {\rm Cost}(h_{\theta}(x),y)=-\log(1-h_{\theta}(x))$ if $y=0$. In the same was as above, I do understand why it's correct, however why is it the only form? For example, why couldn't $|h_{\theta(x)}-y|$ be a good choice for the cost function?

    Question: what is so special about the above form of cost function; why cannot we use another form?

I would appreciate if you could share your understanding of logistic regression.

$\endgroup$
7
$\begingroup$

The logistic regression model is maximum likelihood using the natural parameter (the log-odds ratio) to contrast the relative changes in the risk of the outcome per unit difference in the predictor. This is assuming, of course, a binomial probability model for the outcome. That means that the consistency and robustness properties of logistic regression extend directly from maximum likelihood: robust to missing at random data, root-n consistency, and existence and uniqueness of solutions to estimating equations. This is assuming the solutions are not on the boundaries of parameter space (where log odds ratios are $\pm \infty$). Because logistic regression is maximum likelihood, the loss function is related to the likelihood, since they're equivalent optimization problems.

With quasilikelihood or estimating equations (semiparametric inference), existence, uniqueness properties still hold but the assumption that the mean model holds is not relevant and the inference and standard errors are consistent regardless of model misspecification. So in this case, it's not a matter of whether the sigmoid is the correct function, but one that gives us a trend that we can believe in and is parameterized by parameters that have an extensible interpretation.

The sigmoid, however, is not the only such binary modeling function around. The most commonly contrasted probit function has similar properties. It doesn't estimate log-odds ratios, but functionally they look very similar and tend to give very similar approximations to the exact same thing. One need not use boundness properties in the mean model function either. Simply using a log curve with a binomial variance function gives relative risk regression, an identity link with binomial variance gives additive risk models. All this is determined by the user. The popularity of logistic regression is, sadly, why it's so commonly used. However, I have my reasons (the ones that I stated) why I think it's well justified for it's use in most binary outcome modeling circumstances.

In the inference world, for rare outcomes, the odds ratio can be roughly interpreted as a "relative risk", i.e. a "percent relative change in the risk of outcome comparing X+1 to X". This isn't always the case and, in general, an odds ratio cannot and should not be interpreted as such. However, that parameters have interpretation and can be easily communicated to other researchers is an important point, something sadly missing from the machine learnists' didactic materials.

The logistic regression model also provides the conceptual foundations for more sophisticated approaches such as hierarchical modeling, as well as mixed modelling and conditional likelihood approaches which are consistent and robust to exponentially growing numbers of nuisance parameters. GLMMs and conditional logistic regression are very important concepts in high dimensional statistics.

$\endgroup$
  • 1
    $\begingroup$ Thank you very much for your answer! It seems like I have a huge lack in background. $\endgroup$ – user16168 Sep 29 '13 at 19:52
  • $\begingroup$ I think McCullough and Nelder's book Generalized Linear Models would be a great background resource for a more statistics perspective. $\endgroup$ – AdamO Sep 29 '13 at 22:16
  • $\begingroup$ In general, what textbook do you advise in Machine learning with very detailed descriptive content? $\endgroup$ – user16168 Sep 30 '13 at 14:23
  • $\begingroup$ Elements of Statistical Learning by Hastie, Tibshirani, Friedman. $\endgroup$ – AdamO Sep 30 '13 at 16:47
  • 2
    $\begingroup$ @user48956 Statistical Analysis with Missing Dada, Little & Rubin 2nd ed. Missing data is not "represented" per se, but "handled" by omission. This is not particular to logistic regression: it is the naive approach used by all statistical models. When data are formatted in a rectangular array, rows with missing values are omitted. This is known as a complete case analysis. GLMs and GLMMS are robust to missing data in the sense that complete case analyses are usually unbiased and not very inefficient. $\endgroup$ – AdamO Jul 6 '16 at 20:39
6
$\begingroup$

One way to think about logistic regression is as a threshold response model. In these models, you have a binary dependent variable, $Y$, which is influenced by the values of a vector of independent variables $X$. The dependent variable $Y$ can only take on the values 0 and 1, so you can't model the dependence of $Y$ on $X$ with a typical linear regression equation like $Y_i=X_i\beta+\epsilon_i$. But we really, really like linear equations. Or, at least, I do.

To model this situation, we introduce an unobservable, latent variable $Y^*$, and we say that $Y$ goes from equaling 0 to equaling 1 when $Y^*$ crosses a threshold: \begin{align} Y^*_i &= X_i \beta + \epsilon_i\\ &\\ Y_i &= 0 \;\textrm{if}\; Y_i^*<0\\ Y_i &= 1 \; \textrm{if} \; Y_i^*>0 \end{align} As I have written it, the threshold is at 0. This is an illusion, however. Generally, the model includes an intercept (i.e. one of the columns of $X$ is a column of 1s). This allows the threshold to be anything.

To motivate this model, think of killing bugs with a nerve-toxin pesticide. $Y^*$ is how many nerve cells are killed, and $X$ includes the dose of pesticide delivered to some bug. $Y$ is then 1 if the insect dies and 0 if it lives. That is, if enough nerve cells are killed (and $Y^*$ crosses the threshold), then the bug dies. This is not actually how neurotoxic pesticide work, by the way, but it's fun to pretend.

So, you get a linear regression equation you can't see and a binary outcome you can see. The parameters, $\beta$ are usually estimated via maximum likelihood. If $\epsilon$ is distributed with symmetric distribution function $F$, then $P\{Y_i=1\}=F(X_i\beta)$. Just as you say, you can use any symmetric distribution function you want.

Actually, you can use an asymmetric distribution function if you like, it just makes the algebra a tiny bit harder, as $P\{Y_i=1\}=1-F(-X_i\beta)$.

Now, the distribution function you pick for $\epsilon$ affects your estimation results. The two most common choices for $F$ are normal (yielding the probit model) and logistic (yielding the logit model). These two distributions are so similar that there are rarely important differences in the results between them. Since logit has a very convenient closed form for both cdf and density functions, it's usually easier to use it rather than probit.

Again, just as you say, you could pick any distribution function for $F$ and which one you pick will affect your results.

$\endgroup$
  • $\begingroup$ What you described is exactly the motivation for the probit model, not logistic regression. $\endgroup$ – AdamO Sep 26 '13 at 22:39
  • 6
    $\begingroup$ @AdamO, if the $\epsilon_i$ have a logistic distribution, then this describes logistic regression. $\endgroup$ – Macro Sep 26 '13 at 22:50
  • $\begingroup$ That seems like a very sensitive assumption and one that would be difficult to test. I think logistic regression can be motivated when such error distributions don't hold. $\endgroup$ – AdamO Sep 27 '13 at 16:52
  • 2
    $\begingroup$ @AdamO, however you motivate logistic regression, it's still mathematically equivalent to a thresholded linear regression model where the errors have a logistic distribution. I agree that this assumption may be hard to test but it's there regardless of how you motivate the problem. I recall a previous answer on CV (I can't place it right now) that showed with a simulation study that trying to tell whether a logistic or probit model "fit better" was basically a coin flip, regardless of the true data generating model. I suspect logistic is more popular because of the convenient interpretation. $\endgroup$ – Macro Sep 27 '13 at 17:22
  • 2
    $\begingroup$ @AdamO This is a manifestation of the usual economist/statistician divide, but . . . I don't think logistic regression is semi-parametric. The statistical model is $P(Y_i=1)=\frac{exp(X_i\beta)}{1+exp(X_i\beta)}$. That's parametric. One can (and I do) interpret it as coming from a threshold model with logistic error. If I get worried about making too many assumptions on the error term, I am going to drop logistic regression, not the threshold model. Threshold models can be estimated with much weaker assumptions on the error terms using maximum score and related estimators, for example. $\endgroup$ – Bill Sep 27 '13 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.