I'm confused about logistic regression cost function. What if in some test data is bad prediction? Then one of the terms in sum is equal to infinity and sum of anything with one infinity member is equal to infinity. It means, that the cost function is also equal to infinity. Here is the cost function:enter image description here

I have an idea and I can't find any proof. The idea is to start with weights equal to zero, then all logarithms have parameters equal to 0.5, then log(0.5) is not infinity and you can run gradient descent or whatever to minimize the cost. Am I right?

  • 2
    $\begingroup$ What do you mean "bad prediction"? How can a single prediction lead to infinity? $\endgroup$ – HelloWorld Feb 19 '17 at 12:00
  • 1
    $\begingroup$ I admit I don't really quite understand the question. $\endgroup$ – HelloWorld Feb 19 '17 at 12:01

By "bad prediction" do you mean e.g. $y^{(i)} = 1$ and $h(x^{(i)}) = 0$? That would indeed lead to an infinite loss – but luckily, for any finite weight vector, $h$ is never going to actually achieve values 0 or 1, and $\log h$ will always be finite.

So, you can initialize logistic regression wherever you want. Because the problem is convex, you'll always get an equivalent answer.

The only way that $J$ can actually become infinite is if the problem is linearly separable, in which case driving all the predictions to exactly 0 and 1 becomes the optimal solution. This is usually avoided via regularization.

  • $\begingroup$ Yes, by bad predicition is meant what you said. And you've also answered my question. You are right, that in sigmoid function I never get infinity. When I was looking for answer on my own I tried programming it in Octave and when I start with initial weights = 1 on my dataset, then the cost function was sum of NaN and Inf values. But that's another question, tha one I asked here is answered, thank you :) $\endgroup$ – Vladan Feb 19 '17 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.