The (training) data contains 1280 observations with 1415 features. The test set has additional 380 observations. The data is sparse, that is, many of the variables has many zeros and few positive values. a typical variable\feature distributes with zeros in about 90% of the observations and small tail of few positive values The response is dichotomous, i.e zero-one classification.
I have tried to fit:
- L1,L2 and mixtures of both regularized binomial(logistic) models with
- A simple
glm()model with 100 first selected variables after selecting them with stepwise forward procedure
surprisingly the second fit (hardthresholded model) preformed better on the training and test sets, measuring to goodness of fit with auc (area under the curve in the ROC curve) and misclassification error rate. It even preformed better than a model with first 100 variables selected by L1 regulation.(any idea why?)
For this reason and because of some regulatory conditions in the industry I would like to continue preform variable selection similar to stepwise algorithm but faster.