I have genomic data that I want to play with. I have 3 CSV files, each with ~300 samples and ~19000 genes, which contain information on expression levels, relative expression levels (tumor / normal), and Boolean values indication mutations. I also have an additional CSV file that has ground truth information (when did the patient's cancer progress, how long were they observed).

It makes sense for me to use one of the data CSV files along with the ground truth file to create a supervised learning algorithm. But how would I incorporate the other CSV files? To me, I know that the data for gene A in patient B in datafile C is different than gene A in patient B in datafile D, but they're both gene A, and I am not sure how to distinguish the use of gene A in different contexts in the code I write.

My initial thought is to append something like _1 or _2 to the genes in the other files, combine all the features to have a 300x57000 matrix, and do stuff on that, but that seems like the wrong way to go about it. Is there a name for what I am asking for? Is there a better way to do this? I am having trouble wrapping my head around how this would work

Edit: Another idea I just had was to find a way to combine the data from each file so that each feature has a tuple of values associated with it, and maybe I can throw that into my random forest?

Edit: tl;dr: Feature X has n values associated with it, which may or may not be related. How does this factor into how I use them in my learning algorithm?

  • $\begingroup$ If you are just asking for Python functions / code, that would be off topic here, but I think there is a viable statistical question in this. $\endgroup$ Jan 5, 2017 at 0:34
  • $\begingroup$ I think it's conceptual (I think if I get the concept I can find the relevant function myself) Basically: Feature X has n values associated with it, which may or may not be related. How do I handle this? Or even simpler, what is the name of this? I don't think it's multivariate, or any of the other "multi" prefix'd words I've come across. I don't think this is a super unique situation either — I feel as if this is something that has a standard way of being handled. $\endgroup$
    – Aru Singh
    Jan 5, 2017 at 0:35
  • $\begingroup$ Do the 3 files represent replicate determinations of the same genes on the same patients, or is the relation among the files more complicated? That would play an important part in a useful answer. $\endgroup$
    – EdM
    Jan 5, 2017 at 1:16
  • $\begingroup$ I might not be interpreting the term "replicate determinations" correctly — but I interpret your question as asking if file 1 2 and 3 represent the same thing being measured, but replicates of that measurement. If that is what you are asking, the answer is no; each file has different data. In particular. data on: absolute expression levels, relative expression levels (normal cell/tumor), and Boolean on mutations. The relation is more complicated. $\endgroup$
    – Aru Singh
    Jan 5, 2017 at 1:48

1 Answer 1


Each of the 19000 genes has three characteristics you are considering: expression level, tumor/normal expression ratio, and presence of mutation. If you think that these characteristics are related in ways that might be important for your project, then some initial exploration of those characteristics (without reference to the outcome data) seems in order.

Exploration of distributions of predictors, and relations among predictors, is a standard way to start to reduce dimensionality in the scenario of $p\gg n$ (many more predictors than cases). The example in ISLR of using random forests to relate gene expression to predict cancer types (pages 320-321) only considered the 500 genes (among 20000 genes total) with the largest variance in expression among the samples. Pre-clustering several associated predictors into individual predictors is another accepted approach to this problem; see, for example, section 4.7 of Harrell's Regression Modeling Strategies.

In your particular application the task might be fairly simple. Unless your cancers are of a highly hypermutated type (e.g., patients with DNA polymerase deficiencies or microsatellite instability), very few of the 19000 genes are likely to have mutations in more than 1 or 2 of your 300 individuals. If you restrict your list of mutated genes to those that are mutated in more than, say, 2% of individuals then almost all of the data columns on mutation status can be eliminated.

Typically gene expression data are normalized in some way among the samples (to account for differences in sample mass, assay conditions, etc.) although it's not clear from your question whether that has been done. If not, standard normalization among samples should be done first.

Once normalized gene expression data from tumors is available, it's not clear what additional information the tumor/normal expression ratio will add. If you take the standard approach of analyzing tumor gene expression and tumor/normal expression ratio data in log scales, then the log tumor/normal expression ratio is just the difference between the normalized log expression in the tumor (which you already have) and the normalized log expression in whatever normal tissue was analyzed. So working with the ratio is pretty much equivalent to working with the normalized normal-tissue expression instead. If you don't think that normal-tissue expression levels per se matter to outcome, then you might not need to include the ratio information in your model.

After the initial dimension reduction, however, you will still face the challenges of $p \gg n$. Random forests provide one way to approach this, but intelligent paring down of your number of predictors first, based on subject-matter knowledge, is highly preferable to just throwing all the data into a random forest.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.