I have 199 data, 77-diagnosed and 122-healthy. Each data has 12076 feature (gene expression). When I split data into train and test (N/2 for train and N/2 for test) and run feature selection on train-data and select best features and then run the same feature selection on all the data and select features, the selected features in two situation are so different(for example in the train data 259 features are selected and in the all data 119, and the number of common features is 28). The difference between the two categories is not impressive, and I don't know how to separate them. I tried information gain, random forest, chi-squared, sub-lasso and cfs (from FSelector package in R), but none of them works out. I need to reduce the features and prevent over-fitting and use a classifier to discriminate two classes, I tried SVM as classifier and also KNN (I know it is for clustering but all the results are desperating so I just tried it). The PC1-PC2 plot of data is also very bad (data points are not separable at all). When I divide people in dataset based on the age and sex, the PC1-PC2 plot is very good, but the results remains disappointing (for example when I choose male under 9, the data size reduces to 41 and I think that's the problem). The accuracy after classification is about 65 - 70%. Can anyone guide me how to make it better?
-
$\begingroup$ Did you randomly select the training half? $\endgroup$– Michael R. ChernickJun 29, 2017 at 23:01
-
$\begingroup$ @MichaelChernick I select them randomly but with equal number of every class $\endgroup$– user137927Jun 30, 2017 at 7:19
-
$\begingroup$ Are the features all binary? Does each of the 199 data points have a distinct/unique gene expression? Do you have any known factors? (sounds like age and gender are good to use) $\endgroup$– probabilityislogicJul 1, 2017 at 1:53
-
$\begingroup$ @probabilityislogic All the gene expressions are continuous, because there are 12076 dimension (gene expression) and they are continuous, yes they are distinct and unique (all together, but not every gene expression is unique). We also have factors (gender and age), I use them and they can discriminate very good, but when I shrink data, the results of classification is not good at all. $\endgroup$– user137927Jul 1, 2017 at 9:19
1 Answer
First, if you achieve an accuracy exceeding $65\%$, there is a sense in which that is a solid score. After all, you are beating a baseline model that predicts the majority category every time (which would have accuracy around $61\% = 122/199$). See my discussion here on this kind of comparison, which means that your accuracy between $65\%$ and $70\%$ has an $R^2_{accuracy}$, as I define it in the link, between $0.095$ and $0.225$. While these might not seem high, given the inherent variability in biological data and the possible determinants of health that are excluded from your model (diet, exercise, smoking, etc), these might be pretty good scores. (Indeed, it is routine for papers in the top journals of many fields to have $R^2$ scores like these, so why not $R^2_{accuracy}$ scores like those?)
Second, welcome to the instability of feature selection. This is expected behavior. Indeed, I addressed a similar question just two weeks ago. Frank Harrell of Vanderbilt University has a keynote presentation on YouTube (Why R? 2020) that does some simulations showing this instability, along with a quote I like.
Parsimony is the enemy of predictive discrimination.