As a followup to this question, how does scikit-learn implementation of Lasso (and coordinate_descent algorithm) uses the tol
parameter in practice?
More precisely, in the documentation, we can see:
tol: float, optional
The tolerance for the optimization: if the updates are smaller than tol, the optimization code checks the dual gap for optimality and continues until it is smaller than tol.
Additionally, when the model does not converge, we can get:
ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 8.058438499079784, tolerance: 5.712111291830755 positive)
So:
- How is the duality gap defined in the case of Lasso (/ElasticNet)?
- Why the displayed tolerance in the example above is
5.712111291830755
whereas it was set as0.0001
(default value) in the model? - In pratice, what does
the optimization code checks the dual gap for optimality and continues until it is smaller than tol.
mean?