I'm trying to find a model that will help predict the outcome of the dependent variable (binary). My test subset has 6,000 observations My variables are as follow:

  • 6 categorical (with over 30 levels)
  • 20 continuous
  • 45 continuous (percentage)

I've tried many things for variable selection:

  • PROC Logistic with Stepwise
  • PROC Logistic with Stepwise followed by an all-subset research
  • PROC Hpgenselect with the Stepwise method (select sl, choose SBC or AIC)
  • PROC Hpgenselect with the Lasso method

No matter which method, I rarely get more than 5-6 variables selected, and if I try to score my test subset, I don't get anything better than 55% of correctly classified prediction.

I've also tried by adding modified variables (square, log) and by adding interaction terms.

Is there anything I'm missing?

  • $\begingroup$ Things may not be as bad as they seem. How does this 55% correct classification compare with the proportion based on the actual dependent variable? In other words, if you treat predicted (yes/no) vs actual (yes/no) as a two-way table, how does it breakout into false positives and negatives? $\endgroup$ – DJohnson Nov 18 '16 at 2:40
  • $\begingroup$ Around 40% for both false positive and false negatives. $\endgroup$ – YH_90 Nov 19 '16 at 2:40
  • $\begingroup$ Are you able to post the full two-way table? $\endgroup$ – DJohnson Nov 19 '16 at 14:37

Firstly, try a model-building strategy for which you run all the univariate model regressions, and then combine variables whose p-values were less than 0.2 or 0.25 (during univariate) into a mutiple-variable model.

Also, maybe consider collapsing all of the continuous variables along with the percentages down to e.g. 10 dimensions using PCA. When done with PCA, input the 10 top principal components (new variables with zero correlation, which are standard normal-distributed with mean zero and variance unity). You could recode the categorical variables to their k-1 dummy indicators, and try adding them with the 10 PCs.

Taken together, all of your input predictors could essentially be viewed as a sheer mess by the logistic model. That is, the combination of range and scale of all the predictors could have high correlations (i.e., redundant data) and be all over the map regarding dependency with the binary outcome variable. Correlated predictors in regression models force the variance-covariance matrix (inverse of the Hessian matrix) to be more non-positive definite so that it becomes semi-positive definite with zero eigenvalues.

  • $\begingroup$ When I do the PCA, there's only 1 principal component, which as a loading of 1 on one variable, and all the others are 0 or very close to 0. $\endgroup$ – YH_90 Nov 17 '16 at 20:13
  • $\begingroup$ If you have 65 variables based on continuous and percentile-variables, you should run PCA on the 65x65 correlation matrix, which will be done by SAS almost by default on the data matrix. (Make sure the covariance matrix is not used for PCA). There will be 65 eigenvalues obtained from the correlation matrix. Can you report those here? $\endgroup$ – JoleT Nov 18 '16 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.