Ablation Study training until when?

Question

Given that deep learning models can take weeks to train, having to train 20 different models becomes infeasible when performing an ablation study. Are there some formally accepted common practices on how to conduct an ablation study? (Given a reputable reference)

My main concern is not being able to train until significant convergence (e.g. the last 5% improvement) until the very end that can usually take ~50% of the training time.

Answer 1

There are no common practices for ablation experiments that are widely accepted by the community and this remains an open-problem at the time of writing.

This is a subject I have been actively working on and published a library for PyTorch ablation experiments that you can find here: https://github.com/fostiropoulos/ablator