0
$\begingroup$

I have been searching for articles on feature selection and cross-validation, but could not find enough answer to my specific question. I am currently working on an in-house feature selection pipeline for resting-state fMRI data. I use nested CV structure so that I do feature selection in the inner fold. My pipeline is as follows: 

for each fold in outer fold
  for each fold in inner fold
    run f-test on train data. 
    rank features according to their f-scores.
    apply k-best algorithm with different *k* values (previously sampled from uniform random distribution)
    fit model on train data with k-best features, and test on test data. 
  return CV score for each *k* values. 
  ?????
end

As you see above, my pipeline basically consists of optimizing k parameter in the k-best feature selection. My question is what to do after feature selection. I have two strategies on that (but I am open for any useful and efficient ideas), which are:

1) Following feature selection, I fit the model on train data in the outer fold with k-best features, which are selected in the inner CV fold.

2) Let's say the first 100 features are found to be the best features with the highest CV score. However, as I rank features by their f-scores, different CV folds may give different features found in the first 100 features. So, to get the best features with a high possibility to have useful information, I take union of features and fit the model on train data in the outer fold with those features.

I am not sure which strategy is better.

Improve this question
$\endgroup$
2
  • $\begingroup$ What exactly are you trying to achieve here? Is this about variable selection? $\endgroup$
    – user2974951
    Jul 31, 2019 at 6:24
  • $\begingroup$ @user2974951 Yes, sort of. What I am currently trying to do is to find edges (features, in this case) that have useful information for classifying individuals from two groups (test vs. control). I said edges because I use graph theory to construct network on resting-state fMRI data. $\endgroup$
    – afarensis
    Jul 31, 2019 at 20:06

1 Answer 1

Reset to default
1
$\begingroup$

Let's first sort things a bit:

train_k_features = function (train, k){
    run f-test on train data. 
    rank features according to their f-scores.
    apply k-best algorithm with different *k* values (previously sampled from uniform random distribution)
    fit model on train data with k-best features, and test on test data. 

    return model
}

with that, the inner cross validation becomes more concise:

for each fold in outer fold

  for each fold in inner fold
      for each k in your range of *k*
          model = train_k_features (inner train, k)
  return CV score for all *k* values. 

  select best k
  outer_model = train_k_features (outer train, k)

return CV scores for outer models, all ks chosen for the outer models 

check whether optimization went well (is k stable across the outer CV models?)

fit final model

Inside the outer CV you're now fitting models with automated selection of the optimal k.

We can put the innards of the outer cross validation into a function again: 

train_tuned_model = function (data)
  generate inner folds: splits data -> inner train, inner test

  for each fold in inner fold
      for each k in your *k*
          model = train_k_features (inner train, k)
  return CV score for all *k* values. 

  select best k

  return train_k_features (data, k)

with that the outer cross validation becomes:

for each fold in outer fold

  outer_model = train_tuned_model (outer train)

return CV scores for outer models and all ks chosen for the outer models 

check whether optimization went well (is k stable across the outer CV models?)

final_model = train_tuned_model (data)
Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your helpful answer. What do you recommend for measuring the stability of the optimization? Also, what should I do if k is not stable across the outer CV models? $\endgroup$
    – afarensis
    Aug 2, 2019 at 0:29
  • $\begingroup$ k is not stable across the outer CV models: the purpose of the outer CV is verification/validation of the final model. Thus, you are not allowed any more to do anything to the modeling. Instead, you report that a range of k (possibly also the distribution) was observed and that this may indicate instability of the optimization procedure. $\endgroup$
    – cbeleites unhappy with SX
    Aug 2, 2019 at 11:55
  • $\begingroup$ How to measure stability: depending on the type of model you use, you may be able to formulate stability in the fitted model parameters. If that's not possible (or not sensible), you can use stability of predictions, e.g. std. of all predictions for the same case in repeated/iterated cross validation. Also, it is impossible to comment on how stable k (and the actually chosen features) should be without knowing the data generation process. $\endgroup$
    – cbeleites unhappy with SX
    Aug 2, 2019 at 11:59
  • $\begingroup$ I get the idea that within the outer CV, I validate the final model with feature selected in the inner CV, and I can do anything else than simply showing the distribution of the k across folds. However, I do not still get how to fit the final model. Which k should I use to fit the final model? $\endgroup$
    – afarensis
    Aug 8, 2019 at 2:43
  • $\begingroup$ The fitting of the final model determines its own k inside train_tuned_model. So for the final model, we have an inner CV (for autotuning) but no outer CV (as the final model is trained outside the outer = validation/verification CV). You can then check (but not change) that k and compare whether it differs from the ks found during outer CV. $\endgroup$
    – cbeleites unhappy with SX
    Aug 8, 2019 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.