2
$\begingroup$

If you do linear regression without regularization on close-to-singular data matrix $X$ (or it does not have enough data), the problem arises even in closed-form solution $w = (X^TX)^{-1}X^Ty$ when you do $(X^TX)^{-1}$.

If you do logistic regression, your gradient would look like: $\frac{\partial L(w)}{\partial w} = X(y-\text{logit}(w^TX))$ and I don't see any particular reason why it should diverge if $X$ has not enough data or data is perfectly separable. I would expect gradient to skyrocket when problem is ill-posed, but that is not obviously follows from gradient.

How one can see from analytical derivations that logistic regression would experience serious troubles if we don't have enough data? (same way as be observe inverse of singular matrix in linear regression)

$\endgroup$
  • 2
    $\begingroup$ What is your question? $\endgroup$ – whuber Jan 21 '17 at 0:21
  • $\begingroup$ @whuber my question is: how one can see from analytical derivations that logistic regression would experience serious troubles if we don't have enough data $\endgroup$ – MInner Jan 21 '17 at 1:26
  • 1
    $\begingroup$ Try taking the second derivative, which you need to perform Newton's method to solve for your weights. Your singularity issue lies in there I believe. $\endgroup$ – David Kozak Jan 21 '17 at 1:57
  • $\begingroup$ @DavidKozak that is true; if you do Iterative Reweighed Least Squares, certain things emerge; is there any way to look at it from first-order perspective? In case of linear regression, we somehow manage to see issues even without looking at Hessian? $\endgroup$ – MInner Jan 21 '17 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.