I would like to address a specific machine learning procedure I would like to implement in R with the package caret, which is quite challenging regarding its limitations/possible solutions. In detail, through a feature selection methodology we have developed in my lab, i have acquired 36 features/genes regarding a microarray dataset concerning a binary outcome (disease status: cancer samples and control samples).

Then, my initial goal is to use only these selected features to train a classifier based on this microarray dataset-training set—and then validate it in external datasets, in order to evaluate and initially test the discriminatory power of these features. My main questions are the following:

1)My first initial concern, is due to the relatively small sample size of my training set (60 samples), is still worth to train a classifier and then test it in any datasets  ? In the context of even having identified any significant signatures for my putative classifiers, the results would be underestimated in my external datasets? Or it is worth trying and report any metrics in the prediction for each dataset?

2. Or a more “appropriate” approach, would be to focus on my training set, perform for instance a 10-fold cross-validation and report the test error rates for each fold?

3)On this context, if I follow the first approach: I should first on my training set perform an “exhaustive” cross validation (for instance 10 fold cross-validation with caret repeated a number of times), in order to select some tuning parameters? but also at the same time repeat the total process a different time of random seeds (with different values of random.seed function), and then somehow report the average prediction metrics for each external dataset  ?

(* just to mention that these 36 genes are also part of a much bigger differentially expressed DE list concerning my initial dataset)

I would like to address a specific machine learning procedure I would like to implement in R with the package caret, which is quite challenging regarding its limitations/possible solutions. In detail, through a feature selection methodology we have developed in my lab, I have acquired $36$ features/genes regarding a microarray dataset concerning a binary outcome (disease status: cancer samples and control samples).

Then, my initial goal is to use only these selected features to train a classifier based on this microarray dataset-training set—and then validate it in external datasets, in order to evaluate and initially test the discriminatory power of these features. My main questions are the following:

1) My first initial concern, is due to the relatively small sample size of my training set ($60$ samples), is it still worth it to train a classifier and then test it in any datasets? In the context of even having identified any significant signatures for my putative classifiers, the results would be underestimated in my external datasets? Or it is worth trying and report any metrics in the prediction for each dataset?

2) Or a more “appropriate” approach, would be to focus on my training set, perform for instance a $10$-fold cross-validation and report the test error rates for each fold?

3) On this context, if I follow the first approach: I should first on my training set perform an “exhaustive” cross validation (for instance $10$ fold cross-validation with caret repeated a number of times), in order to select some tuning parameters? but also at the same time repeat the total process a different time of random seeds (with different values of random.seed function), and then somehow report the average prediction metrics for each external dataset?

(* Just to mention that these $36$ genes are also part of a much bigger differentially expressed DE list concerning my initial dataset.)

I would like to address a specific machine learning procedure I would like to implement in R with the package caret, which is quite challenging regarding its limitations/possible solutions. In detail, through a feature selection methodology we have developed in my lab, i have acquired 36 features/genes regarding a microarray dataset concerning a binary outcome (disease status: cancer samples and control samples).

Then, my initial goal is to use only these selected features to train a classifier based on this microarray dataset-training set—and then validate it in external datasets, in order to evaluate and initially test the discriminatory power of these features. My main questions are the following:

1)My first initial concern, is due to the relatively small sample size of my training set, is still worth to train a classifier and then test it in any datasets ? In the context of even having identified any significant signatures for my putative classifiers, the results would be underestimated in my external datasets? Or it is worth trying and report any metrics in the prediction for each dataset?

2. Or a more “appropriate” approach, would be to focus on my training set, perform for instance a 10-fold cross-validation and report the test error rates for each fold?

3)On this context, if I follow the first approach: I should first on my training set perform an “exhaustive” cross validation (for instance 10 fold cross-validation with caret repeated a number of times), in order to select some tuning parameters? but also at the same time repeat the total process a different time of random seeds (with different values of random.seed function), and then somehow report the average prediction metrics for each external dataset ?

(* just to mention that these 36 genes are also part of a much bigger differentially expressed DE list concerning my initial dataset)

I would like to address a specific machine learning procedure I would like to implement in R with the package caret, which is quite challenging regarding its limitations/possible solutions. In detail, through a feature selection methodology we have developed in my lab, i have acquired 36 features/genes regarding a microarray dataset concerning a binary outcome (disease status: cancer samples and control samples).

Then, my initial goal is to use only these selected features to train a classifier based on this microarray dataset-training set—and then validate it in external datasets, in order to evaluate and initially test the discriminatory power of these features. My main questions are the following:

1)My first initial concern, is due to the relatively small sample size of my training set (60 samples), is still worth to train a classifier and then test it in any datasets ? In the context of even having identified any significant signatures for my putative classifiers, the results would be underestimated in my external datasets? Or it is worth trying and report any metrics in the prediction for each dataset?

2. Or a more “appropriate” approach, would be to focus on my training set, perform for instance a 10-fold cross-validation and report the test error rates for each fold?

3)On this context, if I follow the first approach: I should first on my training set perform an “exhaustive” cross validation (for instance 10 fold cross-validation with caret repeated a number of times), in order to select some tuning parameters? but also at the same time repeat the total process a different time of random seeds (with different values of random.seed function), and then somehow report the average prediction metrics for each external dataset ?

(* just to mention that these 36 genes are also part of a much bigger differentially expressed DE list concerning my initial dataset)