I have a dataset of 93 records and 45 radiomics variables from various CT scans. I wanted to check if age and sex could be classified by the variables so I made a new variable with both sex and age. I tried various approaches (PCA, ICA and VIF for feature selection and recursive partitioning for classification, plus elastic net for selection and classification). I am now evaluating the performance of my models, but I have a doubt. Should I perform feature selection on the whole dataset or only on the train data?

  • $\begingroup$ Feature selection, if done, is done only on the training set. $\endgroup$ – user2974951 Sep 14 '18 at 12:49
  • $\begingroup$ @user2974951 ok...so I do feature selection plus the classification on the training set plus the classification, and then I only evaluate how the performance of classification are on the test set. Is that correct? $\endgroup$ – schrodingercat Sep 14 '18 at 12:53
  • 1
    $\begingroup$ Some people would divide their data in 3 parts, one for training, one for feature selection, and one for testing. Sometimes the second step is omitted. You may have a little trouble doing this since you have a small sample, only 93 records. $\endgroup$ – user2974951 Sep 14 '18 at 12:54
  • $\begingroup$ @user2974951 I'll split the dataset only in two parts, then. Thank you! $\endgroup$ – schrodingercat Sep 14 '18 at 16:25

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.