I am trying to predict a behavioral variable using neuroimaging data using supporting vector regression. Since there are ~ 400.000 voxels (=features) in an image and I have a limited sample size I have decided to perform a features selection step. In particular I calculate the univariate correlation between each feature and the dependent variable N times with sample N-1 and I take the lowest estimate of the correlation in order to select only those feature who are stably (across subject) associated with the dependent variable.
In order to select the hyper parameters of the SVR (v and C) I am performing a nested cross validation.
Right now, the whole process looks like this
For every subject in N Take N-1 sample Perform features selection on N-1 For every combination of hyper parameters For every subject in N-1 Fit the model on N-1-1 Test the model on the inner left out subject Chose the best combination of hyper parameters Fit a model on N-1 using the best combination Test the model on the outer left out subject
What I am wondering now is about the feature selection. Is it correct to perform it only one time before the inner loop for the cross-validation of the hyper parameters, or should it be performed within the inner loop, together with the choice of the hyper parameters ? From one point of view, the feature selection is indeed independent from the test sample, but on the other hand it is not cross-validated for the inner sample.
Any take on this ?