When it comes to selection of a subset of variables from a big data matrix, such as a gene expression matrix, I have come across some publications that have picked a somewhat arbitrary approach to select for the most informative variables, e.g. filtering data based on N top percent of variance, coefficient of variation, median absolute deviation, or standard deviation.

  1. Which one of these terms mentioned above can provide a less biased selection and is statistically more significant (specially in gene expression data?)

  2. I have seen such filtering criteria (based on variance) as below in a paper (link) but I can not figure out why these filtering steps have been applied. For example, wouldn't it be enough to select a subset of variable only based on n top percent of IQR?

"To enrich highly informative probe sets, we applied three filters: probe sets had to be among the top 2% of all probe sets with regard to their variance, their maximum minus median expression and their 90% quantile minus median expression across all specimens."

Since I am not an statistician I would appreciate someone could explain these in simple words.

  • 1
    $\begingroup$ I suspect coefficient of variance to be a typo for coefficient of variation. If not, your source is unreliable. As for your question, maximum, median and variance are defined in just about every introductory text. At that level, the 90% quantile is perhaps more commonly called the 90% percentile. Maximum, median and percentiles are expressed in terms of the original units, so you can look at their differences. (Not a biologist, so I don't know what gene expression is precisely. Not a statistician either.) $\endgroup$ – Nick Cox Jan 26 '18 at 9:22
  • 1
    $\begingroup$ The issue is substantive, I imagine, that IQR would not use any of the detailed information above the upper quartile. $\endgroup$ – Nick Cox Jan 26 '18 at 9:24
  • $\begingroup$ @nickCox thanks for mentioning the typo. I am aware of definitions of the terms, but what I don't understand is that what statistical power they have add by doing selection based on those 3 criteria. why not, for example, only top 2 % of most variable genes? PS: gene expression matrix is relative measures of several thousands of genes under investigation across individuals (patients) in an experiments. $\endgroup$ – symo Jan 26 '18 at 10:43
  • $\begingroup$ I think you're talking about someone's scientific judgement here. Either someone explains why in your literature, or 2% is just being offered as what supposedly works well. There is no statistical argument behind 2% that follows from definitions of quantiles or variance. Note that there is a strict statistical meaning to power within statistical inference; I sense you're using the word informally and without wanting or needing to define it, which is fine by me, but it's best to avoid ambiguity. $\endgroup$ – Nick Cox Jan 26 '18 at 11:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.