I have 33 variables my dataset, I need to omit some less significant features then, which is the "best suitable feature selection method " for the Ordinal Logistic Regression?

  • 2
    $\begingroup$ What is the goal of this model? $\endgroup$ Commented Aug 21, 2023 at 17:46

2 Answers 2


I think you can try Factor Analysis, which is similar to PCA. Any feature/ regressor with absolute value of loading <0.1 you can drop. Check out the following link (Python code included): https://www.earthinversion.com/geophysics/exploratory-factor-analysis/

The python documentation of the package can be found here: https://factor-analyzer.readthedocs.io/en/latest/factor_analyzer.html#module-factor_analyzer.factor_analyzer

Few extra things to consider:

  1. Loading Threshold: The suggestion to drop features with absolute loading values below 0.1 is a reasonable starting point. However, the choice of threshold can be somewhat arbitrary and might need some tuning based on the specifics of your dataset and the goals of your analysis. It's good to acknowledge that a balance needs to be struck between discarding potentially important information and keeping the model interpretable and parsimonious.

  2. Consider Alternative Methods: While Factor Analysis can be a good starting point, it's worth mentioning that there are other feature selection methods that might also be suitable for ordinal logistic regression. For example, Lasso (L1 regularization) and Recursive Feature Elimination (RFE) are commonly used techniques that take into account the relationship between features and the dependent variable.

If your aim is to identify variables that significantly contribute to explaining the variability of a dependent variable, you should focus on techniques that are designed for that purpose, such as Lasso, RFE, or other similar methods and if you want to identify latent factors that explain the observed correlations between a set of independent variables Factor Analysis is useful. It's often used in situations where you have a set of observed variables that are believed to be influenced by underlying, unobserved factors, which are not tied to a specific dependent variable, and the main goal is to uncover the common patterns of variation among the observed variables.

  • In some problems, data scientists use Logistic regression to decide what are the important features(Independent variables) that have higher effect on the target(Dependent variable).

    Because logistic regression itself will map variables to variable Importance level. Which may help you deciding which features to use in production. you can search for "logistic regression variable importance" to know more about this.

  • You can also use random forests (Decision trees) and visualize what variables are used in early splits, these variables will have higher effect on dependent variable too. then you can pick n features based on your preference.
  • $\begingroup$ I think this link may help you too analyticsvidhya.com/blog/2016/12/… $\endgroup$ Commented Feb 19, 2019 at 11:31
  • $\begingroup$ Thanking you very much Ferdi. As u suggested logistic regression (LR) for feature selection, but problem is that LR is binary classifier whereas Ordinal LR is a multiclass classifier. So Feature selected by LR, are suitable enough for Ordinal LR? $\endgroup$ Commented Feb 20, 2019 at 8:21
  • $\begingroup$ Actually Abdul, I am the one who answered not Ferdi. Fredi just edited my Answer and removed (Best of luck) I wrote to you at the end. anyway I think Yes you can use these feature at any type of models after you knew that they has an affect on your target (dependent variable) Best Of luck Abdul. $\endgroup$ Commented Feb 20, 2019 at 12:44
  • $\begingroup$ Jazakallahu Khayr. But how to utilize logistic regression to know the importance of features (independent variables)? can you pls share any R code for it? I am a novice in this field, your help will be appreciated. $\endgroup$ Commented Feb 21, 2019 at 4:29
  • $\begingroup$ ` library(caret) mydata <- data.frame(y = c(1,0,0,0,1,1), x1 = c(1,1,0,1,0,0), x2 = c(1,1,1,0,0,1), x3 = c(1,0,1,1,0,0)) fit <- glm(y~x1+x2+x3,data=mydata,family=binomial()) summary(fit) varImp(fit, scale = FALSE) ` referring to link $\endgroup$ Commented Feb 21, 2019 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.