4
$\begingroup$

When evaluating the cost of a neural network during training, should I always expect the cost function to converge towards exactly zero?

Without regularization being applied, the cost converging to 0 would be the desirable behavior, right? (forgetting about overfitting for now)

However, if applying regularization (L2 for example), could be that the cost function converges to 0.4, but it still does a great job when evaluated on new data? Or should I always aim for the cost function to converge to zero?

Improve this question
$\endgroup$
3
  • $\begingroup$ With such a high ratio, it means you could lost acceleration, perhaps you are not even at point of local minima of a cost function, but for sure you are not touching global minima. Are you sure gradient is applied correctly? For some models 0.4 is good enough. But goal is always to converge to global minima and even this doesn't guarantee overall success, as there are other factors to think about... like over fit, train test split etc. $\endgroup$
    – Stenga
    Commented Apr 2, 2018 at 18:42
  • $\begingroup$ I wrote 0.4 only as an example. What I mean is, converging to the global minima that you mention, does it mean converging to exactly 0 ? $\endgroup$
    – sdiabr
    Commented Apr 2, 2018 at 19:03
  • $\begingroup$ global minima is aprox 0. the more decimals after zero you get, the better it is... $\endgroup$
    – Stenga
    Commented Apr 2, 2018 at 19:10

1 Answer 1

Reset to default
4
$\begingroup$

No, in general case.

Think about a data set with "irreducible error", e.g., two data points with exactly same feature but different label. The cost cannot be 0, no matter the definition of the cost function and which parameter to use.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Great example. So then the answer to my question "...could be that the cost function converges to 0.4, but it still does a great job when evaluated on new data?" , is a Yes, right? $\endgroup$
    – sdiabr
    Commented Apr 2, 2018 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.