I've trained 17 regression models using a support vector regression machine providing correlations (r) between true and predicted labels and the mean squared error (MSE) for each model. The target variables were questionnaire scores of 17 different questionnaires from subjects whereas the features were voxels of brain activity.
This means that the target variables were not independent as the different questionnaires were filled out by the same subjects.
Within each SVR model, I've performed 16'000 permutations with permuted labels (permuted questionnaire scores, so I've permuted the subjects) to get a null distribution. This leads to 5 models with significant r's and MSEs (p<0.05).
No, if I would like to compare the models, I've to correct for multiple comparisons, right ? Bonferroni (0.05/17) would be too conservative. Would FDR be appropriate ? As I have only 17 tests, would it make sense to correct for multiple comparisons as p < 0.05 would mean that 1 of 20 tests is a potential false positive...or is my thought wrong here ? Furthermore, the within models significance is derived with a conservative approach using permutations...
Thank you for your answers, michael