3
$\begingroup$

I am trying to predict a behavioral variable using neuroimaging data using supporting vector regression. Since there are ~ 400.000 voxels (=features) in an image and I have a limited sample size I have decided to perform a features selection step. In particular I calculate the univariate correlation between each feature and the dependent variable N times with sample N-1 and I take the lowest estimate of the correlation in order to select only those feature who are stably (across subject) associated with the dependent variable.

In order to select the hyper parameters of the SVR (v and C) I am performing a nested cross validation.

Right now, the whole process looks like this

For every subject in N
    Take N-1 sample 
    Perform features selection on N-1
    For every combination of hyper parameters
        For every subject in N-1
            Fit the model on N-1-1
            Test the model on the inner left out subject               
        Chose the best combination of hyper parameters
    Fit a model on N-1 using the best combination
    Test the model on the outer left out subject

What I am wondering now is about the feature selection. Is it correct to perform it only one time before the inner loop for the cross-validation of the hyper parameters, or should it be performed within the inner loop, together with the choice of the hyper parameters ? From one point of view, the feature selection is indeed independent from the test sample, but on the other hand it is not cross-validated for the inner sample.

Any take on this ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

Usually, performing feature selection inside the inner loop would be the safer option.

Think about if you are able to tune your feature selection with certain parameters too - like the amount of correlation you allow, information you preserve, or similar. If you want to optimize those, not doing so in the inner loop would likely leave you with an overly optimistic error estimate (as you don't have a separate inner-loop performance estimation anymore). Therefore doing such things in the inner loop and using the outer loop for the final error estimation would usually be the way to go.

Update: I tried to sketch a workflow that I think should be applicable for your problem, in as few steps as possible (see below, I hope I didn't mess anything up). If you want to check out more details, consider reading one of those papers:

Varma & Simon (2006). "Bias in error estimation when using cross-validation for model selection." BMC Bioinformatics, 7: 91

Cawley & Talbot (2010). "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation." Journal of Machine Learning Research, 11: 2079-2107 

Do data partitioning (train subjects/test subjects)
Do e.g. repeated CV (leave-subject-out-CV) on train data: For every subject in N:
    Leave out N1
        Leave out N2
            Fit all combinations of feature selection parametrization, hyperparameters, etc. on N-N1-N2
            Evaluate and remember performance for all on the inner left out subject N2
            Evaluate and remember performance for all on the outer left out subject N1
Select "best" parametrization from performance of leaving out N2
Report CV model performance from performance of leaving out N1
Train final model from all training data using chosen "best" parametrization
Test final model - double check that model does what it should
Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks a lot for your input. I am just getting scared by the sheer computation size of the thing. $\endgroup$
    – Federico Nemmi
    Jul 19, 2016 at 12:36
  • $\begingroup$ On a second thought, I was re-reading your answer and I am not sure to got it right. If the features selection is performed within the inner loop, that means that for each subject I will have a subset of features that have been selected. Now, I can choose the hyper-parameters that maximize the correlation between prediction and actual data, but then how can I chose the features to use in the outer loop ? $\endgroup$
    – Federico Nemmi
    Jul 19, 2016 at 14:59
  • $\begingroup$ @FedericoNemmi I've updated my answer, tried to provide a sketch workflow, and provided a link to 2 useful papers giving some more details. $\endgroup$
    – geekoverdose
    Jul 20, 2016 at 7:53
  • $\begingroup$ Thanks a lot for your answer and your time. I am just still puzzled. If I understand correctly, I should try out every possible combinations of hyper-parameters and features in the inner loop. But since I start with ~20000 features, all the possible combinations simply become unbearable from a computation view point (without event taking in account that the combination of all possible h-ps). So should I just learn to code in C to quicken thinks out, or am I grossly misunderstanding ? $\endgroup$
    – Federico Nemmi
    Jul 20, 2016 at 9:56
  • $\begingroup$ @FedericoNemmi Yes, only a validation inside the inner and outer loop is a good estimate in the wider sense. You don't have to try every possible feature combination, but each one you consider should in fact be evaluated in the inner loop too. Lets say you evaluate all possible parametrizations with inner+outer loop to obtain a valid estimate, then select the best parametrization from inner loop performance and report the outer loop performance. $\endgroup$
    – geekoverdose
    Jul 20, 2016 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.