I have a $198$-sample dataset containing miRNA types (numerical features) and one categorical feature "Type" with values "Tumor" or "Healthy".

  Index     miRNA1          miRNA2          miRNA3            Type
   1       48421.52        24242.14        23842.1518        Tumor
   2       2757.96         28965.2         7339.57           Healthy
   3       4300.34         52565.07        6981.41           Healthy
           ...             ...             ...
   198     23854.73        24722.28        7611.53           Tumor

Since there are 1584 of these features in total, I need to select the ones that are most influential towards developing a Tumor.

My approach is described below. Is it correct?

The distributions of features are mostly log-normal. I've transformed each feature with a Box-Cox transformation to get approximately normal distributions. I scaled the values with Min-Max scaler to put them in range $[0,1]$.

miRNA1 has $100$ Healthy samples and $98$ Tumor samples. I should make a null hypothesis that Tumor samples have the same values as Healthy samples. I calculate mean and standard deviation for Tumor samples and Healthy samples, calculate the t-score and calculate the p-value, using significance level of $0.05$ and DF in this case is $97$. This is a two-tails test so it is $p$-value $\times 2$. If it's lower than $0.05$ I reject the null hypothesis and consider miRNA1 as a feature that impacts Tumor development, right?

  • $\begingroup$ Sounds good except that the results of the final model will be extremely biased (too large effects, too good inference etc.) $\endgroup$
    – Michael M
    Sep 4 '19 at 7:28
  • $\begingroup$ @MichaelM Thanks for comment, browsed sereval posts here 1 2 3 and indeed bias is a problem in general, will probably use advice from other posts to minimize it's effect. $\endgroup$
    – Alex
    Sep 4 '19 at 12:05

First, what you have is high-dimensional data. This alone poses some problems and you should use a method which was designed for this, which is better suited.

Second, a regular t-test is a bad idea in this case, it is a univariate test - meaning it does not consider multiple variables together and their possible interactions. Also, p-values are not meant to be used for feature selection.

Nonetheless, if you are fixed on a t-test, it would be better to use a permutation test to test for significance, as you have many variables which will lead to some serious corrections, when you adjust your p-values for multiple testing, and you will adjust them, right?

Finally, personally I would use LASSO regression to solve this, which is a better and simpler option, LASSO automatically performs feature selection and it considers all the variables together, rather than one by one.

  • 2
    $\begingroup$ (+1) piggybacking to add: university stats classes often teach that doing some transformations to make data normal will solve your problems, but that is largely not the case. LASSO is a great option here if you need a full model, but another thing statistical genetics/genomics people do in this situation is false-discovery rate control testing, which I would argue is a bit simpler and works better out-of-the-box (though you gotta be careful with the interpretation). $\endgroup$ Sep 4 '19 at 8:00
  • $\begingroup$ @SheridanGrant You are right, I forgot to add a comment about needlessly transforming variables, since a t-test does not require normally distributed data. As for FDR, it is a good approach, but I would still consider LASSO a better option, since it models all the variables together, while FDR is still used in the context of univariate hypothesis testing. $\endgroup$ Sep 4 '19 at 8:12
  • $\begingroup$ @SheridanGrant and user thanks for your input! I wanted to start with t-test because they seem relatively easy to understand and are used in numerous studies in this area. A lot of papers also use ANOVA, pearson correlation, data clustering techniques study Unfortunately none of these papers describe how they carry out these tests so I don't really know how to get started properly. I will consider using lasso and random Forest for this task. 1/2 $\endgroup$
    – Alex
    Sep 4 '19 at 12:22
  • $\begingroup$ The objective is however to firstly reduce the dataset to around 10-20 features and then experiment with different models. That's how most papers in this area do, they use techniques to get dataset of 10 features and then use ANN to build a model (they have microarray data, maybe that's why they use ANN instead of simpler classifiers. To the distribution part: I normalized them since almost every study does that and I'm a begginer so just did it as well, also read that parametirc tests do assume gaussian like dist. (In these studies they use Kolmogorov-Smirnov test and then normlize) 2/2 $\endgroup$
    – Alex
    Sep 4 '19 at 12:22
  • $\begingroup$ @Alex I understand, feature selection is important. However, you should not use p-values to do that, it is a bad idea that has been discussed numerous times on this site, as to why it leads to potentially very bad results. Additionally: 1) the others measures you mentioned also are not used for feature selection except for clustering, 2) I would not use ANN when you have such a small sample size, 3) a lot of people perform normalization, because of a wrong belief that it will improve the results, 4) LASSO or RF is a good option. $\endgroup$ Sep 4 '19 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.