I am comparing multiple regression machine learning algorithms (MLA) for a project. I have been reading Geron's excellent book 'Hands-On Machine Learning with Scikit-Learn & Tensorflow'. He mentions that you should not spend too much time tweeking hyper-parameters first, just get a few good models together, then work on tweeking the hyper-parameters for each one.
So does this mean that I should find the best MLA for my data using the default parameters, discard the bad ones and only focus on tweeking the best model. Is there not a chance that the discarded regressors could out perform the chosen model if their hyper-parameters were tweeked? I am only asking as I am trying to work out the correct order in which KDD processes should be run.
Any advice welcome.