Timeline for How can I use synthetic data to validate my classification model?
Current License: CC BY-SA 3.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Dec 13, 2016 at 1:30 | comment | added | Jason | Yes exactly that is my initial goal !! to show with an initial approach like MFA, which genes and/or PET variables are the top contributors and their possible biological relevance to my cancer under study. And thank you again for pinpointing the second crusial part of analysis. Perhaps, utlizing further external data would be an unbiased way of further validating these topk features, as also other methodologies. | |
| Dec 13, 2016 at 1:20 | comment | added | EdM | It depends on what you do with the "top K variables." If you simply identify them to explain to your audience what they are, that's OK. Do not, however, fall into the trap of taking those K variables and using them as the predictors in a new standard logistic regression or similar analysis. That would almost certainly be over-fit if K>3, and might not repeat well on new cases even if K= 1 or 2 or 3. Your final model should have no more than 3 unpenalized predictors, whether original variables or dimensions in PCA/MFA, or you should penalize predictors with LASSO or ridge regression. | |
| Dec 12, 2016 at 23:38 | comment | added | Jason | Thank you again I have checked the book, but in order to verify your above approach: you mention again over-fitting when using PCA or MFA as an initial feature selection in order to reduce more the features right ? Because i mentioned MFA in order to just analyze simultaneously the 60 samples based on various groups of variables (i.e. genes and PET variables), and identify the top K variables which contribute most to the construction of the first 2 or 3 dimensions (and other subsequent capabilities)-factominer.free.fr/advanced-methods/… | |
| Dec 12, 2016 at 22:40 | comment | added | EdM | Although MFA and PCA incorporate all the original predictor variables, to avoid over-fitting your data you should only use the top 2 or 3 dimensions returned by FMA/PCA as predictors (without reference to the cancer/normal label), unless you impose some penalty as in ridge regression. Repeat the process on multiple bootstrap samples to validate the model-building process. An Introduction to Statistical Learning with Applications in R is one of many good references to consult on the dangers of over-fitting. | |
| Dec 12, 2016 at 22:15 | comment | added | Jason | Dear Edm, actually in my opinion (as i have understood from the paper and R package FactoMineR), MFA is rather a variant of PCA and not some classification method-it is suitable for my analysis, as you can desribe the same observations (i.e. patient samples) with multiple groups of "different variables", so i naively believe that the meaning of "overfitting" is not related in this case. Now,regarding your second part of answer, an interesting correlation of a "composite signature", could perhaps indicate some gene which could be detected along with the PET variable via some experimental tests | |
| Dec 12, 2016 at 21:58 | comment | added | EdM | I don't have experience with rfe; you can check the documentation and code about whether it automatically does bootstrapping or repeated CV on the entire model building process as is recommended. It's unlikely that MFA will avoid the overfitting with only 30 cancer cases. You should consider no more than 2 or 3 predictors unless you use penalization like LASSO or ridge regression. You can look for correlations of PET to genes without including genes in the logistic regression. What practical clinical situation would use both PET and gene expression in a composite signature? | |
| Dec 12, 2016 at 21:16 | comment | added | Jason | topepo.github.io/caret/recursive-feature-elimination.html | |
| Dec 12, 2016 at 21:14 | comment | added | Jason | In parallel, in this point i also believe that Multiple Factor Analysis could handle a larger number of genes and also PET variables, as it balances the influence of each group--already an initial analysis of this approach again ranks as in the top contributors of the dimensions specific PET variables along with specific genes. Regarding the other crusial point of selection-bias with feature selection methodology: because it is an important point, in the tutorial of caret-section 18.2--the rfe function does not perform the external validation-resampling you have mentioned above? | |
| Dec 12, 2016 at 21:08 | comment | added | Jason | Dear EdM, for a weird reason i just saw your updated comments without having some notification, even though i checked every day. Nevertheless, i would like to mention two crusial points on this matter, except you prefer to create a different post-question for this reason: Firstly, regarding your comments about the importance and novelty of PET variables: yes i have completely in mind your suggestions, but the main reason of mixing together the PET variables with our selected genes, is also to find any interesting correlation between any of them-that is the notion "composite signature" ! | |
| Dec 4, 2016 at 19:28 | history | edited | EdM | CC BY-SA 3.0 |
Added text in response to comments
|
| Dec 4, 2016 at 1:23 | comment | added | Jason | Finally, perhaps an alternative way of further feature selection instead of rfe, would be to perform a PCA-maybe an MFA--as i have different "groups" of features representing the same observations-patients. And then, select for instance the top K features that are most correlated with the first dimensions/participate to the construction of the dimensions separating the two classes. But then, if this approach is more unbiased, how i could then evaluate based on the same dataset, these selected features based on some classification/error metrics ? | |
| Dec 4, 2016 at 1:14 | comment | added | Jason | Overall, regarding your crusial point of over-fitting and bias resulting when using feature selection with the known labels (as recursive feature elimination and other models do) and then train the model with these features but with the same data: thus, as my main goal is to evaluate and promote the possible significance of PET features along any other important gene features, my above procedure would not be an initial "appropriate" approach ? Or your above indication: "you would at least have to include that part of the feature selection within each CV fold or bootstrap resample" ? | |
| Dec 4, 2016 at 1:00 | comment | added | Jason | To continue, my notion was to select from all the 20 different iterations only the features that appeared in all of them, to result thus to the 41 final features. Also, i wanted to inspect if along with the genes any PET variables would be selected, in order to have an initial measure of their possible importance. Then, i just used extreme gradient boosting and trained my dataset with the above selected features. Thus firstly, regarding the size of the selected features, you suggest that i could use various sizes in the rfe function, repeat again the same process and see the final outcome ? | |
| Dec 4, 2016 at 0:49 | comment | added | Jason | Dear EdM, thank you for your detailed and comprehensive answers thus far. Because i have not reported all the details, i will try to add any important information i did not mentioned above, in order to also discuss further this matter in some crusial and specific points: firstly, regarding the feature selection procedure: i used as the pool of features, the combination of the 338 DE genes i mentioned along with the 8 PET variables, with 5 repeats of 10-fold cross validation. But i used 20 different random iterations (with 50 the size of the features that should be returned). | |
| Dec 4, 2016 at 0:13 | vote | accept | Jason | ||
| Dec 3, 2016 at 17:53 | history | answered | EdM | CC BY-SA 3.0 |