I am building a simple logistic regression model on 2D data. Here is the input I use.
I built a logistic regression model using this data and it successfully is able to find the discriminating line between these two classes. Here is the separating line found by my program:
Also, just for debugging, I plotted number of iterations vs J(theta), which also looks fine:
Now, confusing part comes into play. I am investigating the cost surface of this model that the logistic regressions is doing the gradient descent on. I am considering three different loss functions. Sum-of-squared loss, hinge-loss, log-loss.
As far as I know, both hinge loss, sum-of-squared loss and log-loss should produce a convex surface. When I look at my surfaces, however, sum-of-squared surface does not look convex to me. Why is that the case?
For convenience, here is the two critical functions from my code:
function retval = J(X, y, theta) M = size(y, 1); % sum of squared error retval = (1/(2*M)) * sum((h(X, theta) - y).^2); endfunction function retval = h (X, theta) % logistic retval = 1 ./ (1+ exp(-1 * (X * theta))); endfunction