Intuition behind logistic regression

Question

Recently I began studying machine learning, however I failed to grasp the intuition behind logistic regression.

The following are the facts about logistic regression that I understand.

1. As the basis for hypothesis we use sigmoid function. I do understand why it's a correct choice, however why it's the only choice I don't understand. Hypothesis represents the probability that the appropriate output is $1$, therefore the domain of our function should be $[0,1]$, this is the only property of sigmoid function I found useful and appropriate here, however many functions satisfy this property. In addition, sigmoid function has a derivative in this form $f(x)(1-f(x))$, but I don't see the utility of this special form in logistic regression.

   Question: what so special about sigmoid function, and why we cannot use any other function with domain $[0,1]$?

2. The cost function consists of two parameters ${\rm Cost}(h_{\theta}(x),y)=-\log(h_{\theta}(x))$ if $y=1, {\rm Cost}(h_{\theta}(x),y)=-\log(1-h_{\theta}(x))$ if $y=0$. In the same was as above, I do understand why it's correct, however why is it the only form? For example, why couldn't $|h_{\theta(x)}-y|$ be a good choice for the cost function?

   Question: what is so special about the above form of cost function; why cannot we use another form?

I would appreciate if you could share your understanding of logistic regression.

Answer 1

The logistic regression model is maximum likelihood using the natural parameter (the log-odds ratio) to contrast the relative changes in the risk of the outcome per unit difference in the predictor. This is assuming, of course, a binomial probability model for the outcome. That means that the consistency and robustness properties of logistic regression extend directly from maximum likelihood: robust to missing at random data, root-n consistency, and existence and uniqueness of solutions to estimating equations. This is assuming the solutions are not on the boundaries of parameter space (where log odds ratios are $\pm \infty$). Because logistic regression is maximum likelihood, the loss function is related to the likelihood, since they're equivalent optimization problems.

With quasilikelihood or estimating equations (semiparametric inference), existence, uniqueness properties still hold but the assumption that the mean model holds is not relevant and the inference and standard errors are consistent regardless of model misspecification. So in this case, it's not a matter of whether the sigmoid is the correct function, but one that gives us a trend that we can believe in and is parameterized by parameters that have an extensible interpretation.

The sigmoid, however, is not the only such binary modeling function around. The most commonly contrasted probit function has similar properties. It doesn't estimate log-odds ratios, but functionally they look very similar and tend to give very similar approximations to the exact same thing. One need not use boundness properties in the mean model function either. Simply using a log curve with a binomial variance function gives relative risk regression, an identity link with binomial variance gives additive risk models. All this is determined by the user. The popularity of logistic regression is, sadly, why it's so commonly used. However, I have my reasons (the ones that I stated) why I think it's well justified for it's use in most binary outcome modeling circumstances.

In the inference world, for rare outcomes, the odds ratio can be roughly interpreted as a "relative risk", i.e. a "percent relative change in the risk of outcome comparing X+1 to X". This isn't always the case and, in general, an odds ratio cannot and should not be interpreted as such. However, that parameters have interpretation and can be easily communicated to other researchers is an important point, something sadly missing from the machine learnists' didactic materials.

The logistic regression model also provides the conceptual foundations for more sophisticated approaches such as hierarchical modeling, as well as mixed modelling and conditional likelihood approaches which are consistent and robust to exponentially growing numbers of nuisance parameters. GLMMs and conditional logistic regression are very important concepts in high dimensional statistics.

Answer 2

One way to think about logistic regression is as a threshold response model. In these models, you have a binary dependent variable, $Y$, which is influenced by the values of a vector of independent variables $X$. The dependent variable $Y$ can only take on the values 0 and 1, so you can't model the dependence of $Y$ on $X$ with a typical linear regression equation like $Y_i=X_i\beta+\epsilon_i$. But we really, really like linear equations. Or, at least, I do.

To model this situation, we introduce an unobservable, latent variable $Y^*$, and we say that $Y$ goes from equaling 0 to equaling 1 when $Y^*$ crosses a threshold: \begin{align} Y^*_i &= X_i \beta + \epsilon_i\\ &\\ Y_i &= 0 \;\textrm{if}\; Y_i^*<0\\ Y_i &= 1 \; \textrm{if} \; Y_i^*>0 \end{align} As I have written it, the threshold is at 0. This is an illusion, however. Generally, the model includes an intercept (i.e. one of the columns of $X$ is a column of 1s). This allows the threshold to be anything.

To motivate this model, think of killing bugs with a nerve-toxin pesticide. $Y^*$ is how many nerve cells are killed, and $X$ includes the dose of pesticide delivered to some bug. $Y$ is then 1 if the insect dies and 0 if it lives. That is, if enough nerve cells are killed (and $Y^*$ crosses the threshold), then the bug dies. This is not actually how neurotoxic pesticide work, by the way, but it's fun to pretend.

So, you get a linear regression equation you can't see and a binary outcome you can see. The parameters, $\beta$ are usually estimated via maximum likelihood. If $\epsilon$ is distributed with symmetric distribution function $F$, then $P\{Y_i=1\}=F(X_i\beta)$. Just as you say, you can use any symmetric distribution function you want.

Actually, you can use an asymmetric distribution function if you like, it just makes the algebra a tiny bit harder, as $P\{Y_i=1\}=1-F(-X_i\beta)$.

Now, the distribution function you pick for $\epsilon$ affects your estimation results. The two most common choices for $F$ are normal (yielding the probit model) and logistic (yielding the logit model). These two distributions are so similar that there are rarely important differences in the results between them. Since logit has a very convenient closed form for both cdf and density functions, it's usually easier to use it rather than probit.

Again, just as you say, you could pick any distribution function for $F$ and which one you pick will affect your results.

Answer 3

Logistic regression is not originally developed by machine learning community but statistics community. There are a lot of probabilistic views behind it. You may search the following terms to get more information: odds, log odds, generalized linear model, binomial link function.

But for machine learning, the goal is slightly different that we only want to get a higher accuracy (minimize the loss function), but not talking about these assumptions and how data is generated.

You are asking very good questions. So, for machine learning, it is common to use other 'sigmoid function' and loss function to do the classification.

See this question for details

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss