3
$\begingroup$

I've been learning about machine learning and one of the methods I recently learned was that of logistic regression for classification problems. For sake of simplicity, let's assume that each data point has only one feature $x_i$ and its response is $y_i \in \{0, 1\}$, so our predictor function would be $$p_c(x)=\frac{1}{1+e^{-c_0-c_1x}}.$$ I understand the point of the logistic regression and what it does. What I don't understand is where it comes from to begin with. I see that the logistic function has many nice properties. Is it possible to come up with the logistic function as (perhaps) the only function for which a series of nice properties hold? If so, what would those properties be, and how could I prove that any such function leads to the logistic regression? If not, what would be the most natural way to derive such a function from first principles?

P.S: I did notice other questions here that are similar, but none of them quite answer what I did ask. I looked at pretty much every stack exchange question related to logistic regression/function I found, including those you sent. I've also read about it on the Elements of Statistical Learning, but there they start from taking the log of the ratio of the probabilities, whose motivation I also struggled with.

Improve this question
$\endgroup$
6
  • $\begingroup$ Are you taking a machine learning/algorithms course? I imagine that this question is answered on some of those courses. $\endgroup$
    – Adam Rubinson
    Commented Sep 18 at 15:10
  • $\begingroup$ @AdamRubinson yes, but the course is not that much mathematical based, so we haven't seen any proofs unfortunately. $\endgroup$
    – Will199
    Commented Sep 18 at 15:16
  • $\begingroup$ @AdamRubinson I looked at pretty much every stack exchange question related to logistic regression/function I found, including those you sent. I've also read about it on the Elements of Statistical Learning, but there they start from taking the log of the ratio of the probabilities, whose motivation I also struggled with. $\endgroup$
    – Will199
    Commented Sep 18 at 15:24
  • $\begingroup$ @AdamRubinson Done. Thank you for the suggestion. $\endgroup$
    – Will199
    Commented Sep 18 at 15:38
  • 1
    $\begingroup$ If you see linear regression as aiming at a model of $\hat y= a +bx$ ($x$ and $\hat y$ taking any real values) then you can see logistic regression as a model based on the logarithm of odds so $\log\left(\frac{\hat p}{1-\hat p}\right) = a +bx$ $\big(x$ and $\log\left(\frac{\hat p}{1-\hat p}\right)$ taking any real values, i.e. modelled probability $\hat p \in (0,1)\big)$. Logistic regression typically uses maximum likelihood estimates as least squares estimates are difficult. $\endgroup$
    – Henry
    Commented Sep 18 at 15:50

1 Answer 1

Reset to default
3
$\begingroup$

One mathematical motivation for the logistic function, and particularly its inverse the logit (log-odds) function $g(x) = \log(x/(1-x))$, is that it's the canonical link for the binomial distribution (e.g. see here or here or here or here). This means in particular that the iterated least squares algorithm based on Fisher scoring is equivalent to Newton's method, so the most efficient algorithm gives us accurate results (see e.g. here for differences between using "observed information" and "expected information" (Fisher scoring) on estimated standard errors of parameters).

Improve this answer
$\endgroup$