Many statistics textbooks emphasise a manual modeling design approach, whereby the practictioner performs exploratory analysis by hand to assess several factors including whether there's any collinearities in the predictors; if any variables need transforming; removing outliers; if any features can be removed from the dataset to reduce the dimensionality; in addition to numerous diagnostics on the error term, such as its variance and distribution.
This is all fine for exploratory analysis where you're trying to build a model that best explains the data, but if your goal is predictive power then such a manual approach seems too time consuming and impractical.
When developing a model to be used to predict future events, the key evaluation measure is how well it can generalise, typically achieved through a resampling method such as cross-validation or bootstrapping. Thus, the effect of any predictor selection or transformation step needs to be evaluated on a holdout set, which would be very time consuming if performed manually over 10-fold cross-validation.
For instance, if you wanted to reduce the number of variables you would need to perform a stepwise procedure to select the optimal predictors for each resampling iteration, as chosen predictors may change depending on the holdout set. An alternative approach would be to manually choose a few subsets of the predictors before any model fitting has taken place so you're not inputting any bias into the procedure, and run cross-validation on models trained on each of these.
The R package
caret allows for data pre-processing to be carried out at each resampling for a fair estimation of predictive strength, however you are limited by the available processing options rather than a full set of diagnostic steps favoured by many statisticians.
Is correct to use an automated model selection process for predictive modeling, whereas a thorough manual model tweaking stage is more appropriate for exploratory modeling?
It feels almost like cheating to setup a large parameter sweep in
caret to obtain a final model without carrying out all the manual steps I see other statisticians use.