# How to validate if factors are significant or not? Or how to choose factors? Example in Logit, svm, Naive Bayes, and Random Forest?

One sentence summary, what is a systematic factor selection method for any of Logit, svm, Naive Bayes, or Random Forest models?

Let's focus on four most common classification methods, logit regression, svm, naive bayes, and random forest.

Can you please explain how to choose factor? How to determine factor significance? And how to do dimensional reduction? Hints, illustrations, or example on those models, or on your model will be appreciated!

To my knowledge, below are my thought:

stepwise, AIC, BIC, bootstrap, PCA, R^2 etc.. these can be used in factor analysis

I am not from a math/stats background, my knowledge about these are in a high degree of fragmentation. That is why I ask this question, wanted to know how professionals work, from a systematic method or systematic thinking.

• "Please introduce your thoughts" is not a suitable question -- it is to oopen ended and invites opinion. You need to ask something more specific May 6 '17 at 7:52
• @Glen_b Is now more specified? May 8 '17 at 19:58
• 1. I have removed some of the problematic phrases and rephrased some others, while trying to leave the basic question intact, and reopened on that basis, though it's still a bit broad. ... 2. Choosing predictor variables for the methods you mention in your title is not what "factor analysis" is about (the word "factor" means something entirely different there). It can conceivably be related but as it stands it doesn't seem to be directly related to your question. May 9 '17 at 5:31

The question is broad covering several general topics so I will give you partial answers. When you have several candidate predictor variables there are several criteria that can be used for subset selection. AIC and BIC are two of them that penalize for using too many parameters (this is to avoid overfitting). You can find several others discussed in many of the posts here. Most of these use prediction accuracy as the criteria. R$^2$ is not good because it will never decrease when additional variables are add. The adjusted R$^2$ fixes that problem but is only a little better. PRESS and Mallow's $C_p$ are two other measures.

Frank Harrell regression book discusses this and he has given his opinions here in a number of posts.

There are others that can be used based on other criteria such as Gunter's best for detecting qualitative interactions. This approach is useful in medical research, particularly clinical trials.

Principal Component Analysis (PCA) finds orthogonal components (linear combinations of variables) that maximize that variance explained in a multivariate data set. Using the first few prinicpal components can explain most of the variability and can be viewed as dimensionality reduction.

The bootstrap can be used for a wide variety of statistical problems.

Some references: On the bootstrap

Bootstrap Methods: A Practitioners Guide 2nd Edition (2007) M. R. Chernick Wiley

Gunter's method and more

Statistical Methods for Dynamic Treatment Regimes: Reinforcement, Causal Inference and Personalized Medicine (2013) B. Chakrahorty and E. E. M. Moodie Springer

Harrell's regression book

Regression Modleing Strategies with Applications to Linear Models, Logistic and Ordinal Regression and S urvival Analysis 2nd Edition (2015) F. Harrell Springer

• Thank you very very much! So, may I ask, say now you have a case, with about 1000 data point, you need to reduce the dimension as well as list out significant factors to build a model, let's say xgboost, even simpler multiple regression. How do you start? From scratch to end. My steps will be 1. data clean. 2. dimension reduction (Here any example will be appreciated). 3. build model 4. check model statistics 5. model validation, include ROC if it is a bi-classification, in sample cross-validation, and out sample back test. Any thing you want to add? Or anything I am missing? Apr 21 '17 at 13:58
• This is a separate question that someone else may be better equipped to answer. Make it a new question. Apr 21 '17 at 14:20
• stats.stackexchange.com/questions/275003/… Apr 21 '17 at 14:49