Probably the simplest model when dealing with a binary classification problem is logistic regression. This relies on fitting a linear regression model to the data and then applying a sigmoid function to the results. So up to my understanding linear regression, here is used to associate a (let's say continuous) independent variable to a continuous dependent one.

My question is:
"Can we use the result of a linear regression for associating the two variables?"
and if so
"Can this association be compared to another, where the dependent variable is continuous?"

By reluctance to the above questions is that when modelling a binary classification problem, we arbitrarily assign numbers to the two outcomes (e.g. "Yes" -> 1, "No" -> 0). But linear regression is not independent of the magnitude of these numbers (i.e. it would produce different results if "Yes" was mapped to 10 and "No" to 1).

  • Is it considered correct, in practice, to base such associations on linear regression?

  • Can they be compared to associations from linear regression models, where the dependent variable is continuous?

  • 1
    $\begingroup$ It's hard to understand your question because your characterizations do not apply to logistic regression: it neither "applies a sigmoid function" to the results of a linear regression nor does it employ a "continuous dependent variable." $\endgroup$
    – whuber
    Sep 8 '18 at 15:05

The usual use case for logistic regression is when your outcome, or dependent variable, is a binary categorical variable. The fact that the integers $0$ and $1$ are associated with the two cases is because the logistic function is mapping the result to probabilities of belonging to the class associated with the integer $1$. If you recall the definition of a binomial random variable, it is the sum of Bernoulli random variables. And Bernoulli random variables associate the value $1$ with a 'success' and the value $0$ with a 'failure.' By designating one of the binary classes as 'success,' that means 'failure' is not belonging to the 'success' class, and by default, belonging to the other class. So the output of logistic regression is the probability of belonging to the class that was arbitrarily associated with 'success,' and thus the integer $1$.

If you were instead to try to apply a standard linear regression to try to predict membership in one of the classes, call it class A - and remember, even though we focus on one class, not being a member of this class means membership in the other, as there are only two classes in the binary case - then you would get results that are most likely nonsensical. Let's say you did simple linear regression on you classification problem, and let's assume you got a slope of $1$ and an intercept of $0$ for the following. Now let's compare two independent variable values : $x = 10$, and $x = 20$. Are you to interpret the results of your regression as $x = 20$ is twice as likely to belong to the class A than $x = 10$? If $x = 40$, is that four times as likely? Asymptotically, as $x \to \infty$ how can you become infinitely more sure that some item belongs to class $A$? Doesn't that just mean the probability of belonging to class A approaches 1? The dependent variable's numerical value doesn't have a sensible interpretation in this case. The other thing to notice is that you can set a threshold for when you decide one class or the other with logistic regression. If the consequences for making a mistake are symmetric, then anything with an output value above $0.5$ gets assigned to class A, and anything below gets assigned to the other class. If the consequences are not symmetric, you could move this threshold value accordingly. How could you sensibly move that threshold value to account for asymmetric penalties when the output values are unbounded? The point of the logistic function is that it is mapping the result to a bounded range which represents the probability of belonging to class A, and again, since there is only one other class for 'not A', the probability of belonging to the other class is just the complementary probability. The choice of assigning $1$ and $0$ is arbitrary - you could switch them, and the interpretation would be the same - the only difference is which class you decided to deem 'success.'

  • $\begingroup$ Wow,thanks for the detailed answer! Unfortunately I don't have enough rep to upvote. $\endgroup$
    – Jcart
    Sep 10 '18 at 22:26
  • $\begingroup$ Does this explanation apply to binary INdependent variables in linear regression as well? For binary dependent variables, you explain that "as x → ∞ how can you become infinitely more sure that some item belongs to class 𝐴?". Could you say the same for binary independent variables: as x → ∞ you're "inventing" new categories that don't make sense? Alternatively, if x = 0.5, the output for y makes no sense. $\endgroup$
    – Tib
    Apr 15 '20 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.