2
$\begingroup$

Although the obvious answer to my question is yes, since caret is a professional, well-known tool, I tend to be skeptical when using implemented functionality from other packages.

So, caret provides function train which gives you the ability to tune your model by providing a searching grid of the parameters. An additional argument of train is the preprocessing method. Let's say I want to train an SVM with a radial kernel and have my data scaled. A way to achieve this is with the following code:

library(caret)
fitControl <- trainControl(method = "repeatedcv",
                       number = 10,
                       repeats = 10)

model<- train(Class ~ ., data = train_xy,
           method = "svmRadial",
            preProc = c('center', 'scale'),
            tuneLength = 8,
            trControl = fitControl)

My question is: when applying center and scale preprocessing on the data, does caret calculate mean and standard deviation based only on the training set (9 folds) and then apply it on the tuning fold? Or does it preprocess the data before assigning examples to training and tuning sets? In my opinion this is data snooping and should be avoided even during model tuning, despite the fact that the accuracy of this step doesn't account for out-of-sample model accuracy.

Sorry, that was a long post for a yes/no answer.

$\endgroup$

closed as off-topic by gung, Greenparker, COOLSerdash, John, user82102 Jul 19 '16 at 20:16

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – gung, Greenparker, COOLSerdash, John, DJohnson
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ If you are only asking about software (eg, caret), that would be off topic here. You would do best to ask on the r-help listserv or contact the maintainers. If you have a substantive question about data snooping, please edit to clarify. $\endgroup$ – gung Jul 19 '16 at 13:53
  • $\begingroup$ @RichardHardy, I have no idea what's 'right' ;-), I just put them in code formatting b/c for me functions and packages are in the same conceptual category. (By extension, I also put software names--R, SAS, etc--in code formatting.) You should probably do it whichever way makes the most sense to you. $\endgroup$ – gung Jul 19 '16 at 17:12
2
$\begingroup$

Yes, you are correct: caret::train applies preprocessing to only the data that should actually be known to the model at this time - hence avoids "data snooping"$^1$. To be precise: in case of 10CV, you correctly concluded that during CV, preprocessing parameters would each time be determined from 9 of the 10 partitions, then applied to the 10th partition. After CV, for the final model, the whole training data is used of course, so there all data is used for preprocessing as well.

$^1$ I just tried to find the associated statement in the caret documentation, but didn't spot it. But it is documented somewhere (if not in the documentation then maybe in the vignette, or one of Max's presentations).

BTW: I agree with @gung that this question should possibly be migrated to e.g. SO.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.