# Best Suitable feature selection method for ordinal logistic regression

I have 33 variables my dataset, I need to omit some less significant features then, which is the "best suitable feature selection method " for the Ordinal Logistic Regression?

• What is the goal of this model? Commented Aug 21, 2023 at 17:46

I think you can try Factor Analysis, which is similar to PCA. Any feature/ regressor with absolute value of loading <0.1 you can drop. Check out the following link (Python code included): https://www.earthinversion.com/geophysics/exploratory-factor-analysis/

The python documentation of the package can be found here: https://factor-analyzer.readthedocs.io/en/latest/factor_analyzer.html#module-factor_analyzer.factor_analyzer

Few extra things to consider:

1. Loading Threshold: The suggestion to drop features with absolute loading values below 0.1 is a reasonable starting point. However, the choice of threshold can be somewhat arbitrary and might need some tuning based on the specifics of your dataset and the goals of your analysis. It's good to acknowledge that a balance needs to be struck between discarding potentially important information and keeping the model interpretable and parsimonious.

2. Consider Alternative Methods: While Factor Analysis can be a good starting point, it's worth mentioning that there are other feature selection methods that might also be suitable for ordinal logistic regression. For example, Lasso (L1 regularization) and Recursive Feature Elimination (RFE) are commonly used techniques that take into account the relationship between features and the dependent variable.

If your aim is to identify variables that significantly contribute to explaining the variability of a dependent variable, you should focus on techniques that are designed for that purpose, such as Lasso, RFE, or other similar methods and if you want to identify latent factors that explain the observed correlations between a set of independent variables Factor Analysis is useful. It's often used in situations where you have a set of observed variables that are believed to be influenced by underlying, unobserved factors, which are not tied to a specific dependent variable, and the main goal is to uncover the common patterns of variation among the observed variables.

• In some problems, data scientists use Logistic regression to decide what are the important features(Independent variables) that have higher effect on the target(Dependent variable).

Because logistic regression itself will map variables to variable Importance level. Which may help you deciding which features to use in production. you can search for "logistic regression variable importance" to know more about this.

• You can also use random forests (Decision trees) and visualize what variables are used in early splits, these variables will have higher effect on dependent variable too. then you can pick n features based on your preference.
•  library(caret) mydata <- data.frame(y = c(1,0,0,0,1,1), x1 = c(1,1,0,1,0,0), x2 = c(1,1,1,0,0,1), x3 = c(1,0,1,1,0,0)) fit <- glm(y~x1+x2+x3,data=mydata,family=binomial()) summary(fit) varImp(fit, scale = FALSE)  referring to link Commented Feb 21, 2019 at 8:55