2
$\begingroup$

I want to split my sample into a training and a testing set in order to cross-validate my findings. However, I am not sure in which point in the process I have to do the split.

Here is the order I want to do the analysis:

  1. Descriptive analysis on the whole sample
  2. Split into training and testing set
  3. Correlation analysis only on the training set (kick out highly correlated variables)
  4. Logistic regression only on the training set
  5. Use resulting logistic model on the testing sample and validate

Is this correct?

Thanks for your help in advance

$\endgroup$
1
  • $\begingroup$ You need to have same variables in training and test set. So if you do create or drop any features in step 3, then split the data and training sets afterwards. Then steps 4 and 5 are ok :) good luck $\endgroup$
    – Jan Sila
    Commented Oct 10, 2016 at 8:57

1 Answer 1

2
$\begingroup$

The primary purpose of splitting into training and test sets is to verify how well would your model perform on unseen data, train the model on training set and verify its performance on the test set.

Since you haven't built your logistic regression model until step 4, you need not split the data until you reach that step.

Exploratory analysis and removing redundant data from the dataset Steps 1 to 3 can be performed on the entire dataset.

I would additionally recommend to use n-folds cross-fold validation and use its results for a better understanding of how your model would behave with unseen data. For this, split the dataset into $n$ random folds, then one-by-one set aside each fold, train the model on the rest of the data and then cross-check its performance on the set-aside fold. Evaluate the model using the n performance metrics so obtained.

$\endgroup$
2
  • $\begingroup$ Thanks for the clear and very helpful description. If I want to perform some binning of my scale variables... Do I perform this on the entire sample or just on the training set and apply later the same binning rules to the testing set to have the same variables? $\endgroup$
    – TheRedhead
    Commented Oct 10, 2016 at 11:17
  • $\begingroup$ Once you have decided on binning values, you would apply the binning function using the same bin boundary values to all new data used for the model. So it really does not matter if you form bins before splitting or afterwards. However, when determining bin boundary values, you should use data that is as representative of the population as possible, hence, decide bin sizes based on the entire dataset before splitting into test-train. $\endgroup$ Commented Oct 10, 2016 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.