2
$\begingroup$

In Logistic Regression, the assumption is that the data must be linearly separable, but if the data is not linearly separable then we can't apply Logistic Regression?

$\endgroup$
3

1 Answer 1

5
$\begingroup$

It is actually exactly the opposite: If the data are linearly separable, logistic regression will not converge: the $\beta$s will rise infinitely as the logistic function approaches, but never reaches the form of a step function.

Logistic regression minimises the cost function:

$$ L(Y|X, \beta) = -\sum_i y_i \ln p_i + (1-y_i) \ln (1-p_i) $$

There is no closed form solution to this and the minimisation has to be performed numerically. Here,

$$ p_i = \frac{1}{1+\text{e}^{-\beta x_i}} $$

is the probability of belonging to the class labeled as "1". It is modelled by the logistic function (hence logistic regression), which is bound to $(0, 1)$:

Logistic function for different betas

That means that its logarithm is always negative, going towards $-\infty$ as its argument approaches $0$. The above cost function would reach its minimum, zero, if the arguments of $\ln$, $p_i$ and $(1-p_i)$, for classes labeled "1" and "0", respectively, could be one. For that to happen, the exponential term in the denominator would need to be either exactly $0$ (for the class labeled "1") or $+\infty$ (for "0"), and for that to happen, the vector $\beta$ would need to have infinite components. Since infinity can never be reached numerically, no numerical algorithm can converge.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.