20
$\begingroup$

I'm doing exploratory data analysis (EDA) on a dataset. Then I will select some features to predict a dependent variable.

The question is:
Should I do the EDA on my training dataset only? Or should I join the training and test datasets together then do the EDA on them both and select the features based on this analysis?

$\endgroup$

5 Answers 5

11
$\begingroup$

I'd recommend having a look at "7.10.2 The Wrong and Right Way to Do Cross-validation" in http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf.

The authors give an example in which someone does the following:

  1. Screen the predictors: find a subset of “good” predictors that show fairly strong (univariate) correlation with the class labels
  2. Using just this subset of predictors, build a multivariate classifier.
  3. Use cross-validation to estimate the unknown tuning parameters and to estimate the prediction error of the final model

This sounds very similar to doing EDA on all (i.e. training plus test) of your data and using the EDA to select "good" predictors.

The authors explain why this is problematic: the cross-validated error rate will be artificially low, which might mislead you into thinking you've found a good model.

$\endgroup$
0
1
$\begingroup$

So you want to identify independent variables that have an effect on your dependent variable?

Then, both of your approaches are actually not really recommendable.

After having defined your research question, you should develop your theory. That is to say, that using the literature, you should identify variables which should have an effect (you should be able to explain the reason).

$\endgroup$
2
  • 6
    $\begingroup$ While this view seems at heart to agree with the classical concept of statistical tests (and as such it pains me to disagree), there's plenty of modern problems for which this is just not feasible. For example, suppose you want to see if any of the 20,000 protein-coding genes are associated with a new hereditary disease. There is no background that would prepare you, no way to "come up with a theory" and an EDA is only way to get started. And if you have enough data for an EDA and confirmatory analysis, you can actually get somewhere. $\endgroup$
    – Cliff AB
    Jan 7, 2016 at 18:17
  • 3
    $\begingroup$ "you should develop your theory" - that's a good idea, but not always possible, especially, in the industry. Sometimes, you just go ahead with forecasting without developing any theories $\endgroup$
    – Aksakal
    Mar 4, 2016 at 18:22
1
$\begingroup$

Applying EDA on test data is wrong.

Training is the process of looking into the correct answers to create the best model. This process it not just limited to running code on training data. Using information from EDA to decide which model to use, to tweak parameters, and so forth is part of the training process and hence should not be allowed access to test data. So to be true to yourself, use test data only to check your model's performance.

Also, if you realize the model doesn't perform well during testing and then you go back to adjusting your model, then that is not good either. Instead, split your training data into two. Use one for training and another to test and tweak your model(s). See What is the difference between test set and validation set?

$\endgroup$
0
1
$\begingroup$

After the paragraph of this answer. Hastie further explains p.245:

"Here is the correct way to carry out cross-validation in this example:

  1. Divide the samples into K cross-validation folds (groups) at random.
  2. For each fold k = 1, 2, . . . , K
    (a) Find a subset of “good” predictors that show fairly strong (univariate) correlation with the class labels, using all of the samples except those in fold k.
    (b) Using just this subset of predictors, build a multivariate classifier, using all of the samples except those in fold k.
    (c) Use the classifier to predict the class labels for the samples in fold k."
$\endgroup$
-2
$\begingroup$

You do EDA on the entire data set. For instance, if you're using leave-one-out cross validation, how would you do EDA only on a training data set? In this case every observation is training and holdout at least once.

So, no, you form your understanding of the data on the entire sample. If you're in the industrial set up, it's even more evident. You're expected to show the trends and general description of the data to the stakeholders in the firm, and you do that on the entire sample.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.