What exactly is tol (tolerance) used as stopping criteria in sklearn models?

What exactly is the tol (tolerance for stopping criteria) in scikit-learn? What is that quantity which is checked with tol to end the training?

As you noted, tol is the tolerance for the stopping criteria. This tells scikit to stop searching for a minimum (or maximum) once some tolerance is achieved, i.e. once you're close enough. tol will change depending on the objective function being minimized and the algorithm they use to find the minimum, and thus will depend on the model you are fitting. There is no universal tolerance to scikit.

For example, when computing the coefficients for a logistic regression:

tol : float
Stopping criterion. For the newton-cg and lbfgs solvers, the iteration
will stop when max{|g_i | i = 1, ..., n} <= tol
where g_i is the i-th component of the gradient.


For a multilayer perceptron model:

tol : float, optional, default 1e-4
Tolerance for the optimization. When the loss or score is not improving
by at least tol for two consecutive iterations, unless learning_rate
is set to 'adaptive', convergence is considered to be reached and
training stops.


I think the question is not specific enough to be answered.

Scikit-learn is a big library for machine learning with python, different algorithms have different optimization problems. And in most problems tol are used as a stopping criteria for the optimization. BUT in different model / algorithm, the tol can be different.

For example, in Lasso, the documentation says

The tolerance for the optimization: if the updates are smaller than tol, the optimization code checks the dual gap for optimality and continues until it is smaller than tol.

This tol can only be used in Lasso, since it checks "dual gap", if you are training a neural network, the "dual gap" definition does not exist.

On a very high level, many machine learning task can be formulated into a "iterative process" to get all the parameters in the model, but when should we stop the iteration? There are different ways to stop.

• Limit number of iterations
• Check if the parameters converge (do not change over iterations)
• Check some other metrics (such as gradient, gap be beteween primal and dual)
• Many more

When we do checking, tol can be used.