Logistic regression cost function finding multiple minima in matlab using fminunc

Question

Good afternoon,

I am a beginner at machine learning and am trying to implement a basic logistic regression using the formula for the cost function we learned in class:

https://miro.medium.com/max/2836/1*_52kKSp8zWgVTNtnE2eYrg.png (cost function) https://miro.medium.com/max/2964/1*o4Dy1w4n2kDOLA8UEwGC9g.png (hypothesis function)

I implemented the following code, and it looks like depending on my initial guess, fminunc usually finds a local minimum near to the guess, but also sometimes stops for other reasons (such as "undefined at initial point" or "cannot decrease the objective function along the search direction") indicating there are many local minima in addition to potentially other issues. I also have similar issues with fminsearch, and updated the code below with fminsearch. If anyone with more experience could provide some insight, it would be greatly appreciated.

Here is my code:

%logistic regression: predicting benign vs. malignant tumors with data on
%x0=1, size, patient age, and bloodflow rate
clear
global x y;
x=[1,7,50,42;1,77,77,2;1,87,75,1;1,87,70,21;1,60,91,10;1,80,90,3;1,30,20,23;1,10,25,20;1,15,20,33;1,10,20,23;1,50,60,1;1,10,20,15;1,40,70,5;1,10,10,24;1,80,80,1];
y=[0;1;1;1;1;1;0;0;0;0;1;0;1;0;1];
fminsearch(@cost,[-1,.1,.1,.1])


function f=cost(a)
global x y;
f=0;
e=2.71828;
for i=1:length(y)
    f=f+(y(i)*log(1./(1+e^(-a*x(i,:)'))));
    f=f+((1-y(i))*log(1-(1./(1+e^(-a*x(i,:)')))));
end
f=-f./length(y);
end
```

Answer 1

The maximum likelihood problem is not guaranteed to have a solution as the so-called "perfect separation of classes" problem may arise. In that case, the log-likelihood is unbounded. This is very likely to happen with small samples such as yours, in which the number of parameters is not much smaller than the number of observations.

A second issue may arise: a poor initialization of the parameter vector makes the exponentials in the logistic function and their derivatives numerically indistinguishable from 0. This prevents a derivative-based algorithm like fminunc from making any progress. This is the so-called "vanishing gradient problem", well-known in the training of neural nets (logistic regression is just a special case of neural-net training). Setting the parameter vector equal to zero should solve the problem (try neighborhoods of zero to check the sensitivity to starting value). Moreover, you should carefully set the gradient tolerance in the stopping criteria. You could also try derivatives-free optimization algorithms like fminsearch (in my experience the latter often works better than fminunc).

Answer 2

Note that you have 4 features and only 6 data points. It is very likely the perfect separation was happening in your example.

To fix this, you can add more data points (use the whole data instead of just some samples), reduce number of features, or add regularization.

One of my answers have the demo of the problem and why regularization will help.

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?