0
$\begingroup$

I would like to address a specific machine learning procedure I would like to implement in R with the package caret, which is quite challenging regarding its limitations/possible solutions. In detail, through a feature selection methodology we have developed in my lab, I have acquired $36$ features/genes regarding a microarray dataset concerning a binary outcome (disease status: cancer samples and control samples).

Then, my initial goal is to use only these selected features to train a classifier based on this microarray dataset-training set—and then validate it in external datasets, in order to evaluate and initially test the discriminatory power of these features. My main questions are the following:

1) My first initial concern, is due to the relatively small sample size of my training set ($60$ samples), is it still worth it to train a classifier and then test it in any datasets? In the context of even having identified any significant signatures for my putative classifiers, the results would be underestimated in my external datasets? Or it is worth trying and report any metrics in the prediction for each dataset?

2) Or a more “appropriate” approach, would be to focus on my training set, perform for instance a $10$-fold cross-validation and report the test error rates for each fold?

3) On this context, if I follow the first approach: I should first on my training set perform an “exhaustive” cross validation (for instance $10$ fold cross-validation with caret repeated a number of times), in order to select some tuning parameters? but also at the same time repeat the total process a different time of random seeds (with different values of random.seed function), and then somehow report the average prediction metrics for each external dataset?

(* Just to mention that these $36$ genes are also part of a much bigger differentially expressed DE list concerning my initial dataset.)

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you tell us what the actual sample size is? $\endgroup$
    – Hugh
    Commented Nov 16, 2016 at 22:36
  • $\begingroup$ Dear Hugh, please excuse me for not mention it above-totally 60 samples in my training set-30 samples in each of the two classes $\endgroup$
    – Jason
    Commented Nov 16, 2016 at 22:47

1 Answer 1

Reset to default
0
$\begingroup$

Since the predictors are all genes I presume that means they are all binary variables (either true or false depending on whether a person has that gene).

First of all you can do some Principle Component Analysis (PCA) on your data to remove some features which add little to the overall variance in the data. This technique captures the most variation for a given number of features so it will maximize the information you have in a set number of features.

This will transform your features from binary to continuous (the validity of this is discussed in the top answer to this question).

The golden rule is to start simple and work towards a more complicated model. You only want a classifier not an inference of what is causing a true/false classification so this opens up your model possibilities greatly. Logistic regression is a basic model which you can start with (and also allows for inference). Perhaps after that you can ask a follow up question with more details of your progress.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Dear Hugh, maybe i have described above something wrong-i mentioned that these are gene expression data, that is i have already perfomed a feature selection, and all these 36 are continuous variables. Moreover, my prediction of interest is a categorical variable, that is the Disease status i mentioned (binary, two levels) $\endgroup$
    – Jason
    Commented Nov 16, 2016 at 23:43
  • 1
    $\begingroup$ Ok I missed that you had done feature selection. As well as logistic regression another model which works well for few samples is a naive bayes classifier. You can read about their advantages on this webpage blog.echen.me/2011/04/27/choosing-a-machine-learning-classifier $\endgroup$
    – Hugh
    Commented Nov 17, 2016 at 0:23
  • $\begingroup$ thank you for this useful post. i have also considered various other models, like exteme gradient boosting--however, my main issue is regardless of the classifier chosen, if i should stick on evaluation metrics regarding my training dataset, rather than moving also to the independent blind datasets $\endgroup$
    – Jason
    Commented Nov 17, 2016 at 0:31
  • 1
    $\begingroup$ You should only use the independent blind dataset as a final test of your model, not as a way to guide model selection. If you train 5 types of models with your training data and test them on the independent data to select the best one then you're artificially inflating the performance on the independent data. You need some data which was never used before in order to do a fair test on the final model you produce. $\endgroup$
    – Hugh
    Commented Nov 17, 2016 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.