What exactly is tol (tolerance) used as stopping criteria in sklearn models?

Question

What exactly is the tol (tolerance for stopping criteria) in scikit-learn? What is that quantity which is checked with tol to end the training?

Answer 1

As you noted, tol is the tolerance for the stopping criteria. This tells scikit to stop searching for a minimum (or maximum) once some tolerance is achieved, i.e. once you're close enough. tol will change depending on the objective function being minimized and the algorithm they use to find the minimum, and thus will depend on the model you are fitting. There is no universal tolerance to scikit.

For example, when computing the coefficients for a logistic regression:

tol : float
     Stopping criterion. For the newton-cg and lbfgs solvers, the iteration
     will stop when ``max{|g_i | i = 1, ..., n} <= tol``
     where ``g_i`` is the i-th component of the gradient.

For a multilayer perceptron model:

tol : float, optional, default 1e-4
     Tolerance for the optimization. When the loss or score is not improving
     by at least tol for two consecutive iterations, unless `learning_rate`
     is set to 'adaptive', convergence is considered to be reached and
     training stops.

Answer 2

I think the question is not specific enough to be answered.

Scikit-learn is a big library for machine learning with python, different algorithms have different optimization problems. And in most problems tol are used as a stopping criteria for the optimization. BUT in different model / algorithm, the tol can be different.

For example, in Lasso, the documentation says

The tolerance for the optimization: if the updates are smaller than tol, the optimization code checks the dual gap for optimality and continues until it is smaller than tol.

This tol can only be used in Lasso, since it checks "dual gap", if you are training a neural network, the "dual gap" definition does not exist.

---

On a very high level, many machine learning task can be formulated into a "iterative process" to get all the parameters in the model, but when should we stop the iteration? There are different ways to stop.

• Limit number of iterations
• Check if the parameters converge (do not change over iterations)
• Check some other metrics (such as gradient, gap be beteween primal and dual)
• Many more

When we do checking, tol can be used.