3
$\begingroup$

Good afternoon,

I am a beginner at machine learning and am trying to implement a basic logistic regression using the formula for the cost function we learned in class:

https://miro.medium.com/max/2836/1*_52kKSp8zWgVTNtnE2eYrg.png (cost function) https://miro.medium.com/max/2964/1*o4Dy1w4n2kDOLA8UEwGC9g.png (hypothesis function)

I implemented the following code, and it looks like depending on my initial guess, fminunc usually finds a local minimum near to the guess, but also sometimes stops for other reasons (such as "undefined at initial point" or "cannot decrease the objective function along the search direction") indicating there are many local minima in addition to potentially other issues. I also have similar issues with fminsearch, and updated the code below with fminsearch. If anyone with more experience could provide some insight, it would be greatly appreciated.

Here is my code:

%logistic regression: predicting benign vs. malignant tumors with data on
%x0=1, size, patient age, and bloodflow rate
clear
global x y;
x=[1,7,50,42;1,77,77,2;1,87,75,1;1,87,70,21;1,60,91,10;1,80,90,3;1,30,20,23;1,10,25,20;1,15,20,33;1,10,20,23;1,50,60,1;1,10,20,15;1,40,70,5;1,10,10,24;1,80,80,1];
y=[0;1;1;1;1;1;0;0;0;0;1;0;1;0;1];
fminsearch(@cost,[-1,.1,.1,.1])


function f=cost(a)
global x y;
f=0;
e=2.71828;
for i=1:length(y)
    f=f+(y(i)*log(1./(1+e^(-a*x(i,:)'))));
    f=f+((1-y(i))*log(1-(1./(1+e^(-a*x(i,:)')))));
end
f=-f./length(y);
end
```
$\endgroup$
2
$\begingroup$

The maximum likelihood problem is not guaranteed to have a solution as the so-called "perfect separation of classes" problem may arise. In that case, the log-likelihood is unbounded. This is very likely to happen with small samples such as yours, in which the number of parameters is not much smaller than the number of observations.

A second issue may arise: a poor initialization of the parameter vector makes the exponentials in the logistic function and their derivatives numerically indistinguishable from 0. This prevents a derivative-based algorithm like fminunc from making any progress. This is the so-called "vanishing gradient problem", well-known in the training of neural nets (logistic regression is just a special case of neural-net training). Setting the parameter vector equal to zero should solve the problem (try neighborhoods of zero to check the sensitivity to starting value). Moreover, you should carefully set the gradient tolerance in the stopping criteria. You could also try derivatives-free optimization algorithms like fminsearch (in my experience the latter often works better than fminunc).

$\endgroup$
6
  • $\begingroup$ Thank you, what would be an appropriate number of observations to avoid this problem? I have increased the set to 15 observations with 3 parameters (plus x0=1 as the zeroth parameter) but I am still having the same issue. $\endgroup$
    – Derek
    Jun 26 at 23:22
  • $\begingroup$ Could you update the code in the question with the new observations? $\endgroup$
    – user4422
    Jun 27 at 0:35
  • $\begingroup$ Also check the second part I added to the answer $\endgroup$
    – user4422
    Jun 27 at 1:10
  • $\begingroup$ Yes, I have updated the code with the new observations. Also, I have the same problem with fminsearch currently with extreme sensitivity to initial guess and also sometimes ending the search due to maximum number of evaluations for fminsearch exceeded. $\endgroup$
    – Derek
    Jun 28 at 0:04
  • $\begingroup$ I ran your code after you added new data, and you still have perfect separation of classes: compare the class predictions from the estimated model with the true ones, and you will see that they coincide. $\endgroup$
    – user4422
    Jun 28 at 8:18
2
$\begingroup$

Note that you have 4 features and only 6 data points. It is very likely the perfect separation was happening in your example.

To fix this, you can add more data points (use the whole data instead of just some samples), reduce number of features, or add regularization.

One of my answers have the demo of the problem and why regularization will help.

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.