How can I use synthetic data to validate my classification model?

Question

Through R and based on a microarray gene expression dataset (60 samples in total-30 cancer and 30 control samples) and R package caret, i have performed a feature selection regarding a binary categorical outcome (Disease status). My final selected subset, is comprised of both gene features as also clinical continuous variables (the initial dataset, was produced by merging and batch effect corrected 2 affymetrix microarray datasets with similar phenotype condition, and also paired-each patient has both cancer and control samples-so in total 30 patients: 30 cancer and 30 adjucent control samples). Regarding the feature selection methodology, briefly the initial pool of features utilized was the DE gene list from my comparison of cancer versus control samples. Afterwards, an initial feature selection took place with a methodology we have implemented in our lab, resulting in 338 genes. Then, i performed a recursive feature elimination with these genes and the 9 continuous clinical variables i mentioned above, which gave 41 final "composite" features.

Moreover, except from a simple initial inspection of my combined composite feature set with cross-validation, i would like also somehow to perform an initial validation in an independent dataset, in the way of testing the classifier trained in my initial dataset with these features. The major problem of simply selecting a microarray dataset from GEO and/or other repositories, is that these PET features, have been only measured in the same patients that also the microarrays have been produced (an important novelty that i would like somehow to test).So, i could not have any external samples or datasets with these clinical features.

Thus, there any package or methodology that i could implement in R, in order to perform a possible simulation of my above dataset with only these 41 features, and then utilize this "synthetic" dataset for external validation with the classifier constructed in my initial training/analyzed dataset ? If not, what are my alternatives for validating this model?

(*Just to mention, the Disease status in the same phenotype information in all my dataset: that is colorectal adenocarcinoma samples or normal samples)

Answer 1

There are 3 statistical (rather than coding) questions here. First is whether or how to use synthetic data as external validation for your classification scheme. As my answer is "you shouldn't," the second is how you can properly validate the scheme based on the data that you do have. The third is whether, in a data set with 30 positive cases, you will be able to validate a model with 41 features.

First, generating a synthetic data set of positron emission tomography (PET) values based on your microarray and other data might be possible, but unless there is a strong theoretical basis for your mapping of microarray and other data to PET values (which seems unlikely) then you could at best use the empirical relations within your present data to generate the synthetic PET data. That does not seem to be a true external validation of your classification scheme as the empirical mapping might depend heavily on the peculiarities of your present data.

Second, it is still possible to do substantial validation of a model, or at least of your model-generation process, based on your data set. Depending on exactly how you used the functions of the caret package, you might already have the necessary information, and you certainly could use the interfaces provided by caret to perform internal validation. This, however, requires careful attention to which parts of your feature selection were based simply on the predictor features and which depended on their relations to known cancer/normal status. That is the topic of this page among many other pages on this site.

You are free to select features for model building based on the features themselves without reference to the known classifications. For example, microarray data analyses typically remove genes with little variance in expression among all samples, as such genes are unlikely to contain any useful information. Unsupervised clustering to find groups of samples with similar expression profiles, without reference to the classification, is a powerful tool. This can provide a single score based on the set of expression values; you can then explore the relation of this score to your classes of interest. You can look for predictors highly correlated to each other (again, without reference to the known "outcome" classes) and choose to remove some or combine them in your set of candidate features. These types of choices do not affect the statistical validity of your later model building.

Once you use the known cancer/normal classes to choose features, however, the assumptions of independence, etc, needed to perform standard statistical tests have been violated. That use of the outcome data to choose features has to be incorporated into model validation. That's why the author of the caret package warns:

One potential issue [is] over-fitting to the predictor set such that the wrapper procedure could focus on nuances of the training data that are not found in future samples (i.e. over-fitting to predictors and samples).

Cross-validation (CV; ideally, with K-fold CV done many times on independent data splits, a good deal more than the "simple initial inspection ... with cross-validation" you have done so far) or bootstrap resampling can be used for evaluating the quality of your model, but the validation must include all the steps, including the feature selection process, that were used in your model building. Again, from the web page on recursive feature elimination in caret:

Since feature selection is part of the model building process, resampling methods (e.g. cross-validation, the bootstrap) should factor in the variability caused by feature selection when calculating performance...improper use of resampling to measure performance will result in models that perform poorly on new samples.

In your case, as your gene set was chosen based on known cancer/normal classes, you would at least have to include that part of the feature selection within each CV fold or bootstrap resample. If PET features were selected based on known classes, that selection would also need to be repeated for each fold or resample. As noted in this answer, the choice of features will probably vary from fold to fold in CV or among bootstrap resamples. That's OK, as what you are validating is not the final model per se but the process you used to build it. You still present the features chosen from the full data set; the cross-validation or bootstrapping provides information about the validity of your approach and the potential optimism of your model when applied to new data.

Third is the issue of having 41 "features" with only 30 cancer cases. Without further explanation, that raises fears of substantial over-fitting. The usual rule of thumb in binary classification is to limit the number of features to 1 per 10-20 cases in the least frequent class: maybe 2 to 3 features for your data set. Although you didn't specify the particular classification method you used during your recursive feature elimination and for your final model, you evidently did not use standard logistic regression. Perhaps you used logistic ridge regression (with its penalization of regression coefficients to reduce the effective number of parameters), or a random forest (which can handle large numbers of correlated predictors). Please be careful in how you present your number of "features"; it seems likely that you have a large number of highly correlated "features." If your model passes appropriate internal validation that's OK, but you need to make sure that you are ready to respond to a reviewer who claims that you have substantially over-fit your data.

Added in response to comments:

Please step back for a moment and focus on your strategy rather than the tactics that you have enquired about in this and other questions on this site.

Your primary goal seems to be to determine whether your PET measures help distinguish cancer from normal tissue. PET data come from imaging on living individuals, while microarray data require biopsies or surgically resected tumor specimens. If you already have a biopsy or tumor specimen, a pathologist would probably be preferred to microarray or PET for the cancer/normal determination. (A pathologist probably provided the ground truth of cancer/normal for your samples). Also there is over a decade of previous work on microarray distinctions between colon cancer and normal tissue. So your novel PET measures, obtained with essentially non-invasive techniques and not requiring intervention by a surgeon, seem to be what's crucial here.

On that basis, your strategy should be to test statistically how well your PET measures specifically work. Data-mining with tools like those accessed via caret may be an appropriate strategy in some applications (like determining useful gene-expression sets), but that strategy (lumping in your PET measures with a large number of microarray values) may be inappropriate for documenting that your PET measures are useful.

Happily, dealing with 9 continuous PET measures on 30 regions each of cancerous and normal tissue is a fairly simple problem. For example, you could perform PCA on those 9 measures over the entire data set (including both normal and cancer) and perform logistic regression, restricting your predictors to the 2 principal components that provide the largest contributions to the variance. That gives you 2 orthogonal predictors for 30 cancer cases, for an acceptable 15 cases per predictor. Or you can use logistic regression methods that penalize individual regression coefficients, like LASSO (which provides variable selection) or ridge regression (essentially a weighted rather than all-or-none principal-components regression, which I have found better behaved in situations with modest numbers of correlated predictors). Internal validation would be done as I noted above, repeating all steps of your modeling process on multiple re-samples (bootstrap or CV) of your data.

Focusing on the novel PET measures would seem to meet both your interests (and those of other cancer researchers, clinicians and patients) better than folding them in with microarray results into a large data-mining operation. That doesn't mean that you should abandon your microarray work, but you should not let the abundance of microarray measurements get in the way of evaluating the value of your PET data, which should be of primary importance. As you are evidently at an academic medical center, there should be local statisticians available to provide close guidance on how to accomplish your goals reliably.