2
$\begingroup$

I have a doubt regarding logistic regression.I know that it separates the data into 2 parts.Is it possible that it leads to a curve as shown in the example of underfitting and overfittingLogistic Regrssion

What i know is that adding more layers(Neural networks) of logistic regression can cause these curves .My question is can a standalone Logistic Regression without any layer lead these types of non linear curves. And if NO,how can we define the case of overfitting and underfitting in that scenario visually.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Yes by 'feature engineering'. If you pass as input, x,y and eg polynomial terms. $\endgroup$
    – seanv507
    Commented Dec 1, 2019 at 17:35
  • 4
    $\begingroup$ Logistic regression does not by itself separate cases into 2 classes. It models the log-odds of class membership. Any separation into predicted class membership depends on a choice of cutoff for predicted probabilities. That cutoff might have a default choice at p=0.5 (log-odds of 0) but that cutoff can be changed based on things like the relative costs of false-positives versus false-negatives. $\endgroup$
    – EdM
    Commented Dec 1, 2019 at 19:51
  • $\begingroup$ See stats.stackexchange.com/questions/127042/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Aug 30, 2020 at 13:25

1 Answer 1

Reset to default
1
$\begingroup$

"Standalone" logistic regression can indeed only draw a straight line, as it defines a linear relationship between the predictors and the log of the odds (our response).

Overfitting can occur in logistic regression in exactly the same way that it can occur for regular linear regression. The reason it's hard to picture it using those diagrams is that you're only visualizing two dimensions, where the idea of over fitting isn't as intuitive.

So linear model overfitting is difficult to visualize precisely because it occurs when you are working in too many dimensions. So this might be side stepping your question, but I think it's actually best to not try and directly visualize it, but just think through an example step by step. Imagine that we're trying to predict someone's weight. We can create a simple linear model that is a decent fit using just their height, and that should generalize fairly well. If we add gender, that should will improve the fit, and also generalize well. But if we also add 26 binary variables based on whether or not they have a given first letter in their last name ("A", "B", "C", etc), well, in a small data set we might improve our fit (maybe the lone "Quinn" in our data set is very heavy, and we capture that), but it probably won't generalize very well (as we don't think there's any real association there). Hence, we are overfitting.

While that was using binary variables (starting letter, gender), it's the exact same concept, and still a linear model (but binary variables can be a little easier to visualize). The fact is, linear models over fit when they involve too many dimensions, relative to the actual sources of variability you're trying to capture in the observed data set. The concept of overfitting in logistic regression is the exact same, the only difference is your target (the log of the odds, not just any continuous response variable).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Of possible interest: datascience.stackexchange.com/a/79994/73930 $\endgroup$
    – Dave
    Commented Aug 30, 2020 at 14:14
  • 1
    $\begingroup$ "Standalone" logistic regression is pretty obsolete. Any continuous predictor can be expanded into a regression spline to allow for any degree of nonlinearity. $\endgroup$
    – Frank Harrell
    Commented Mar 13, 2021 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.