7
$\begingroup$

I'm new to Machine Learning. I'm basically confused about when to perform train test split.

Is the order given below correct?

  1. Split entire data into training and test set

  2. Extract Features from training data

  3. Fit the classification model to the features extracted from training data

  4. Extract the same features, which were computed in step 2, from test data

  5. Apply the fitted model in step 3 to the features extracted from test data in step 4 to evaluate the model

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Yes, your procedure is accurate. It's important keeping in mind that the validation/test set simulates samples that you don't have at the moment of model fitting. For this reason, you never use any information from these samples. $\endgroup$
    – piplustwo
    Commented Aug 18, 2020 at 14:28
  • $\begingroup$ @ping , I asked a question here : stats.stackexchange.com/questions/484292/… Could u plz answer it? $\endgroup$
    – Debbie
    Commented Aug 24, 2020 at 22:23

1 Answer 1

Reset to default
6
$\begingroup$

Your procedure is correct generally. In a more complex loop, additional operations may include validation, hyper-parameter optimisation, feature selection etc.

Typically, feature extraction follows exploratory data analysis (EDA), where you get to know your data, analyse/summarise it, draw intuitive conclusions. In EDA, you don't necessarily do a train/test split.

Note that, if you repeat steps 2-3 in a feedback loop so that you test whether newly extracted features (e.g. interaction variables) are useful for the model or not, you'll need a validation step.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ It's always a good practice doing some sort of validation also in EDA in order to reduce the chances of overfitting. $\endgroup$
    – piplustwo
    Commented Aug 18, 2020 at 14:26
  • 1
    $\begingroup$ Yeah (+1), that also depends on the depth of EDA. That's why I said not necessarily. $\endgroup$
    – gunes
    Commented Aug 18, 2020 at 14:27
  • 1
    $\begingroup$ Agree. EDA is a damn beast (sorry for the technical language)! :-D $\endgroup$
    – piplustwo
    Commented Aug 18, 2020 at 14:28
  • 2
    $\begingroup$ If you do EDA before splitting rather than on the training data only, and you draw any conclusions from it that influence your later classification model, the later evaluation of the model on the test data becomes invalid, because decisions have been made that depend on the test data as well. (This is probably what ping hinted at already.) In many situations this may not make a big difference, but it can be a problem. $\endgroup$
    – Christian Hennig
    Commented Aug 18, 2020 at 14:30
  • 1
    $\begingroup$ This question and answer assumes that there is a very large data set to start with, on the order of several thousands of samples. There's a danger in applying this approach to smaller data sets; see this answer for example. Internal validation might be preferred with smaller data sets. $\endgroup$
    – EdM
    Commented Aug 18, 2020 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.