In Logistic Regression, the assumption is that the data must be linearly separable, but if the data is not linearly separable then we can't apply Logistic Regression?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ I think you are thinking of SVM, logistic regression has no such assumption. $\endgroup$– user2974951Commented Apr 24, 2020 at 11:25
-
1$\begingroup$ medium.com/@akshayc123/logistic-regression-87f7fbb4aaf6 I've read it from here $\endgroup$– Harshit AhluwaliaCommented Apr 24, 2020 at 11:31
-
1$\begingroup$ That's weird, what they are describing looks very much how SVM works, not logistic regression. $\endgroup$– user2974951Commented Apr 24, 2020 at 11:37
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
It is actually exactly the opposite: If the data are linearly separable, logistic regression will not converge: the $\beta$s will rise infinitely as the logistic function approaches, but never reaches the form of a step function.
Logistic regression minimises the cost function:
$$ L(Y|X, \beta) = -\sum_i y_i \ln p_i + (1-y_i) \ln (1-p_i) $$
There is no closed form solution to this and the minimisation has to be performed numerically. Here,
$$ p_i = \frac{1}{1+\text{e}^{-\beta x_i}} $$
is the probability of belonging to the class labeled as "1". It is modelled by the logistic function (hence logistic regression), which is bound to $(0, 1)$:
That means that its logarithm is always negative, going towards $-\infty$ as its argument approaches $0$. The above cost function would reach its minimum, zero, if the arguments of $\ln$, $p_i$ and $(1-p_i)$, for classes labeled "1" and "0", respectively, could be one. For that to happen, the exponential term in the denominator would need to be either exactly $0$ (for the class labeled "1") or $+\infty$ (for "0"), and for that to happen, the vector $\beta$ would need to have infinite components. Since infinity can never be reached numerically, no numerical algorithm can converge.
