Is it better to do exploratory data analysis on the training dataset only?

Question

I'm doing exploratory data analysis (EDA) on a dataset. Then I will select some features to predict a dependent variable.

The question is:
Should I do the EDA on my training dataset only? Or should I join the training and test datasets together then do the EDA on them both and select the features based on this analysis?

Answer 1

I'd recommend having a look at "7.10.2 The Wrong and Right Way to Do Cross-validation" in http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf.

The authors give an example in which someone does the following:

1. Screen the predictors: find a subset of “good” predictors that show fairly strong (univariate) correlation with the class labels
2. Using just this subset of predictors, build a multivariate classifier.
3. Use cross-validation to estimate the unknown tuning parameters and to estimate the prediction error of the final model

This sounds very similar to doing EDA on all (i.e. training plus test) of your data and using the EDA to select "good" predictors.

The authors explain why this is problematic: the cross-validated error rate will be artificially low, which might mislead you into thinking you've found a good model.

Answer 2

Applying EDA on test data is wrong.

Training is the process of looking into the correct answers to create the best model. This process it not just limited to running code on training data. Using information from EDA to decide which model to use, to tweak parameters, and so forth is part of the training process and hence should not be allowed access to test data. So to be true to yourself, use test data only to check your model's performance.

Also, if you realize the model doesn't perform well during testing and then you go back to adjusting your model, then that is not good either. Instead, split your training data into two. Use one for training and another to test and tweak your model(s). See What is the difference between test set and validation set?

Answer 3

So you want to identify independent variables that have an effect on your dependent variable?

Then, both of your approaches are actually not really recommendable.

After having defined your research question, you should develop your theory. That is to say, that using the literature, you should identify variables which should have an effect (you should be able to explain the reason).

Answer 4

After the paragraph of this answer. Hastie further explains p.245:

"Here is the correct way to carry out cross-validation in this example:

1. Divide the samples into K cross-validation folds (groups) at random.
2. For each fold k = 1, 2, . . . , K
   (a) Find a subset of “good” predictors that show fairly strong (univariate) correlation with the class labels, using all of the samples except those in fold k.
   (b) Using just this subset of predictors, build a multivariate classifier, using all of the samples except those in fold k.
   (c) Use the classifier to predict the class labels for the samples in fold k."

Answer 5

You do EDA on the entire data set. For instance, if you're using leave-one-out cross validation, how would you do EDA only on a training data set? In this case every observation is training and holdout at least once.

So, no, you form your understanding of the data on the entire sample. If you're in the industrial set up, it's even more evident. You're expected to show the trends and general description of the data to the stakeholders in the firm, and you do that on the entire sample.