Logistic regression cost surface not convex [duplicate]

Question

I am building a simple logistic regression model on 2D data. Here is the input I use.

I built a logistic regression model using this data and it successfully is able to find the discriminating line between these two classes. Here is the separating line found by my program:

Also, just for debugging, I plotted number of iterations vs J(theta), which also looks fine:

Now, confusing part comes into play. I am investigating the cost surface of this model that the logistic regressions is doing the gradient descent on. I am considering three different loss functions. Sum-of-squared loss, hinge-loss, log-loss.

Hinge loss:

Log-loss:

Sum-of-squared loss:

As far as I know, both hinge loss, sum-of-squared loss and log-loss should produce a convex surface. When I look at my surfaces, however, sum-of-squared surface does not look convex to me. Why is that the case?

For convenience, here is the two critical functions from my code:

function retval = J(X, y, theta)
    M = size(y, 1);
    % sum of squared error
    retval = (1/(2*M)) * sum((h(X, theta) - y).^2); 
endfunction

function retval = h (X, theta)
    % logistic
    retval =  1 ./ (1+ exp(-1 * (X * theta)));
endfunction

Answer 1

Note that, you have a perfect separation problems on logistic regression. This topic is well discussed in CV. Here is one:

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In the link I provided, I also plotted the cost function contour (instead of the surface) for perfect separation case.

I am not checking if your plot is correct or not. Also the shape of the cost function is also heavily depending on data. However, your data is linear separable, logistic regression will not converge. Details can be found in

Regularization methods for logistic regression

For sum of squares loss, even for a data set without perfect seperation, the objective is not convex! Details can be found in this post.

What is happening here, when I use squared loss in logistic regression setting?

Answer 2

I won't be able to do complete justice to your question. one explanation could be the nature of the cost function. The least square function in case of the hypothesis reaching wrong answers gives a unit value. This can lead to many local optima as the cost function tolerates small inclusion of wrong data. but in case of log loss, we have the cost function giving infinity as the output when the hypothesis is nearing the wrong answer. Therefore the likelihood of the cost function having local optima is much lesser.

Hope I answered your query. I would love any addition or suggestions.