5
$\begingroup$

I am building a simple logistic regression model on 2D data. Here is the input I use.

enter image description here

I built a logistic regression model using this data and it successfully is able to find the discriminating line between these two classes. Here is the separating line found by my program:

enter image description here

Also, just for debugging, I plotted number of iterations vs J(theta), which also looks fine:

enter image description here

Now, confusing part comes into play. I am investigating the cost surface of this model that the logistic regressions is doing the gradient descent on. I am considering three different loss functions. Sum-of-squared loss, hinge-loss, log-loss.

Hinge loss:

enter image description here

Log-loss:

enter image description here

Sum-of-squared loss:

enter image description here

As far as I know, both hinge loss, sum-of-squared loss and log-loss should produce a convex surface. When I look at my surfaces, however, sum-of-squared surface does not look convex to me. Why is that the case?

For convenience, here is the two critical functions from my code:

function retval = J(X, y, theta)
    M = size(y, 1);
    % sum of squared error
    retval = (1/(2*M)) * sum((h(X, theta) - y).^2); 
endfunction

function retval = h (X, theta)
    % logistic
    retval =  1 ./ (1+ exp(-1 * (X * theta)));
endfunction
$\endgroup$
  • 1
    $\begingroup$ Why should SSE loss produce a convex curve? Logistic regression is not least squares. $\endgroup$ – AdamO Mar 14 '17 at 15:36
  • 6
    $\begingroup$ It's well-known that SSE is not convex for logistic regression. ccs.neu.edu/home/vip/teach/MLcourse/2_GD_REG_pton_NN/… You can also just show it directly by use of the definition of convexity. $\endgroup$ – Sycorax says Reinstate Monica Mar 14 '17 at 15:41
  • $\begingroup$ Thank you folks. This makes sense now. @Sycorax, can you post an answer so that I can accept? $\endgroup$ – Sait Mar 14 '17 at 15:59
  • $\begingroup$ MLE is convex, but least squares is not for logistic regression. $\endgroup$ – p-value Mar 15 '17 at 1:10
  • $\begingroup$ @Sycorax formal proof can be found in this answer $\endgroup$ – Haitao Du May 9 '18 at 15:28
1
$\begingroup$

Note that, you have a perfect separation problems on logistic regression. This topic is well discussed in CV. Here is one:

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

In the link I provided, I also plotted the cost function contour (instead of the surface) for perfect separation case.

I am not checking if your plot is correct or not. Also the shape of the cost function is also heavily depending on data. However, your data is linear separable, logistic regression will not converge. Details can be found in

Regularization methods for logistic regression

For sum of squares loss, even for a data set without perfect seperation, the objective is not convex! Details can be found in this post.

What is happening here, when I use squared loss in logistic regression setting?

$\endgroup$
0
$\begingroup$

I won't be able to do complete justice to your question. one explanation could be the nature of the cost function. The least square function in case of the hypothesis reaching wrong answers gives a unit value. This can lead to many local optima as the cost function tolerates small inclusion of wrong data. but in case of log loss, we have the cost function giving infinity as the output when the hypothesis is nearing the wrong answer. Therefore the likelihood of the cost function having local optima is much lesser.

Hope I answered your query. I would love any addition or suggestions.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.