Many statistics textbooks emphasise a manual modeling design approach, whereby the practictioner performs exploratory analysis by hand to assess several factors including whether there's any collinearities in the predictors; if any variables need transforming; removing outliers; if any features can be removed from the dataset to reduce the dimensionality; in addition to numerous diagnostics on the error term, such as its variance and distribution.

This is all fine for exploratory analysis where you're trying to build a model that best explains the data, but if your goal is predictive power then such a manual approach seems too time consuming and impractical.

When developing a model to be used to predict future events, the key evaluation measure is how well it can generalise, typically achieved through a resampling method such as cross-validation or bootstrapping. Thus, the effect of any predictor selection or transformation step needs to be evaluated on a holdout set, which would be very time consuming if performed manually over 10-fold cross-validation.

For instance, if you wanted to reduce the number of variables you would need to perform a stepwise procedure to select the optimal predictors for each resampling iteration, as chosen predictors may change depending on the holdout set. An alternative approach would be to manually choose a few subsets of the predictors before any model fitting has taken place so you're not inputting any bias into the procedure, and run cross-validation on models trained on each of these.

The R package caret allows for data pre-processing to be carried out at each resampling for a fair estimation of predictive strength, however you are limited by the available processing options rather than a full set of diagnostic steps favoured by many statisticians.

Is correct to use an automated model selection process for predictive modeling, whereas a thorough manual model tweaking stage is more appropriate for exploratory modeling?

It feels almost like cheating to setup a large parameter sweep in caret to obtain a final model without carrying out all the manual steps I see other statisticians use.

  • $\begingroup$ Textbook advocacy is one thing. Modeling expertise is another. To the extent that your experience permits you to take shortcuts in a textbook prescribed process -- the more you can compress it -- the greater the levels of efficiency you can achieve in doing so. Other important factors include the "costs" of being wrong, overfitting, and so on, versus over-building or spending ridiculous amounts of time on something that doesn't need or deserve it -- I.e., the likely "shelf life" of the model. $\endgroup$ – DJohnson Nov 12 '15 at 16:12
  • $\begingroup$ Thanks @DJohnson, that's a good point. I'm at the stage where I don't have the experience necessary to make informed decisions yet $\endgroup$ – Stuart Lacy Nov 12 '15 at 17:05
  • 1
    $\begingroup$ Don't underestimate yourself. You are at the stage where you can make informed decisions and recommendations. The fact is, it doesn't get all that easier as you progress since you also become aware of more potential pitfalls. $\endgroup$ – DJohnson Nov 12 '15 at 17:23
  • 1
    $\begingroup$ Since you mention explanatory versus predictive modelling, here is a related thread. It will not have an answer to your question, but it could be worth taking a look when it comes to your second-to-last paragraph. $\endgroup$ – Richard Hardy Nov 12 '15 at 17:53
  • $\begingroup$ I was actually prompted to ask this question after reading the Shmueli paper linked in the that question. I'd definitely recommend others confused on this issue to have a read, as it succinctly explains an issue that many statisticians take as a given. $\endgroup$ – Stuart Lacy Nov 13 '15 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.