While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says:

Without regularization, the asymptotic nature of logistic regression i.e (it never quite hit 0 & never quite hit 1) would keep driving loss towards 0 on all examples and never get there, driving the weights for each feature to +infinity or -infinity in high dimensions.

A good example where it holds true is :

Imagine that you assign a unique id to each example, and map each id to its own feature. If you don't specify a regularization function, the model will become completely overfit.

The above in reference to the below loss function: Loss function for Logistic regression

Could anyone help me understand this point via an example or even at intuition level?

  • $\begingroup$ Could you clarify the meaning of "asymptotic"? Presumably this refers to a postulated sequence of datasets, but exactly how do these datasets vary as you go through the sequence? $\endgroup$ – whuber Nov 5 '18 at 13:35
  • $\begingroup$ "Asymptotic" means that the regression values never quite hit 0 & never quite hit 1, i.e (0,1) ! according to the explanation I found in the Google Machine learning course referenced above in the web link, it has nothing to with the features space(dataset), it's only with the logistic regression output value.! $\endgroup$ – Anu Nov 5 '18 at 20:03
  • 1
    $\begingroup$ I'm afraid that's an insufficient description--nor does it appear to agree with conventional meanings of "asymptotic." It is likely that your source has some kind of model of either collecting more and more data or creating more and more "features" or covariates. It's not possible to answer your question definitively without knowing something about that model. $\endgroup$ – whuber Nov 5 '18 at 20:06
  • $\begingroup$ it's a "Linear logistic regression model", Does it help to decipher the confusion? $\endgroup$ – Anu Nov 5 '18 at 20:14
  • $\begingroup$ No, that's not related to the asymptotics. It's a different issue altogether. The intended asymptotics seem to be related to the hypothetical "Imagine that you assign a unique id to each example, and map each id to its own feature." But that's silly--almost any model (whether logistic regression or otherwise) would be overfit because this imagined model supplies a separate parameter for each observation. $\endgroup$ – whuber Nov 5 '18 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.