Methods / approach to improve the predictive accuracy of a logistic regression model

Question

The situation:

I have a logistic model that should predict a defect (1=defect, 0=no defect). My model uses 4 out of 14 parameters, which are significant for my dependent variable (tested through summary() and the anova() chi-squared test). Furthermore, I used 80% (~5000, with ~1200 defects) of my data to train the model and 20% (~1200, ~300 defects) to test it.

My results are:

If the cutoff I use is 50%*

• Sensitivity = 23% (percent of correctly predicted defects/total number of actual defects)
• Specificity = 98% (percent of correctly predicted non defects/total number of non defects)
• Accuracy = 80% (ratio of correctly predicted units/total number of units)

If the cutoff I use is the percent of defects in the training set [i.e., training defects / all training rows (~1200/~5000 = ~24%)]

• Sensitivity = 55%
• Specificity = 81%
• Accuracy = 75%

My question is:

How can I train my model to get a better result for my sensitivity (defect rate)? (Or is there any fault in my approach, i.e. how can i check my model for scientific correctness??)

I'm somehow lost at this point and appreciate any help, reference to a book or link that guides me in the right direction.

Answer 1

The fact that you are using 4 out of 14 parameters implied that you used significance testing to select the variables. This is invalid. There are a number of other problems:

• Your total sample size is far too small for data splitting to be a reliable method
• You are seeking arbitrary cutoffs
• You are not using a proper accuracy score such as deviance, generalized $R^2$, or Brier score; failure to use a proper score results in a bogus model

An accurate model is one whose high-resolution calibration curve (I recommend using lowess for this purpose) is near a 45-degree line. You are likely to need to use data reduction, blinded to $Y$, for reducing the initial dimensionality of the problem, but that depends on whether your signal:noise ratio is small.

Case studies in logistic model building may be found at http://biostat.mc.vanderbilt.edu/rms . And make sure that $Y$ (defect or not) is truly all-or-nothing. If you can measure degree of defect you will have more statistical information and a better model.

Answer 2

Lots could be said here. gung had the best most concise answer. But to add...
(1) If the cutoff I use is 50%, a max of 25% of the defects get detected is this how you want to define the quality of your model? You can detect all the defects if you change your cutoff.
(2) split sample is suboptimal.
(3) prediction is hard
(4) classically the first approach is to see if your model is miss-specified, could be better specified. Should a variable be categorial rather than linear. Is the relationship between the variable and outcome non-linear. Should you consider interaction terms. But none of these will magically make a poorly predicting model suddenly great.

Answer 3

There is no direct answer for this question, you have to try different things. From my experience, you can try different methods to boost your sensitivity:

1- If you have a lot of missing values you could try imputed methods such as KNN.

2- Try different Machine learning algorithms SUCH AS svm AND rf RATHER THAN LOGISTIC REGRESSION

3- If you have large number of features so I suggest you to try different features selection such as LASSO

4- Be careful to tune your parameters

5- If none of the above works, you can try Ensemble methods such as bagging, boosting and stacking. Integrating different models will give you a good boost.

Good luck