3
$\begingroup$

The situation:

I have a logistic model that should predict a defect (1=defect, 0=no defect). My model uses 4 out of 14 parameters, which are significant for my dependent variable (tested through summary() and the anova() chi-squared test). Furthermore, I used 80% (~5000, with ~1200 defects) of my data to train the model and 20% (~1200, ~300 defects) to test it.

My results are:

If the cutoff I use is 50%*

  • Sensitivity = 23% (percent of correctly predicted defects/total number of actual defects)
  • Specificity = 98% (percent of correctly predicted non defects/total number of non defects)
  • Accuracy = 80% (ratio of correctly predicted units/total number of units)

If the cutoff I use is the percent of defects in the training set [i.e., training defects / all training rows (~1200/~5000 = ~24%)]

  • Sensitivity = 55%
  • Specificity = 81%
  • Accuracy = 75%

My question is:

How can I train my model to get a better result for my sensitivity (defect rate)? (Or is there any fault in my approach, i.e. how can i check my model for scientific correctness??)

I'm somehow lost at this point and appreciate any help, reference to a book or link that guides me in the right direction.

$\endgroup$
  • 3
    $\begingroup$ There is always a limit to how accurately you can predict an outcome. There is just only so much information in your data. $\endgroup$ – gung - Reinstate Monica Dec 4 '13 at 16:21
8
$\begingroup$

The fact that you are using 4 out of 14 parameters implied that you used significance testing to select the variables. This is invalid. There are a number of other problems:

  • Your total sample size is far too small for data splitting to be a reliable method
  • You are seeking arbitrary cutoffs
  • You are not using a proper accuracy score such as deviance, generalized $R^2$, or Brier score; failure to use a proper score results in a bogus model

An accurate model is one whose high-resolution calibration curve (I recommend using lowess for this purpose) is near a 45-degree line. You are likely to need to use data reduction, blinded to $Y$, for reducing the initial dimensionality of the problem, but that depends on whether your signal:noise ratio is small.

Case studies in logistic model building may be found at http://biostat.mc.vanderbilt.edu/rms . And make sure that $Y$ (defect or not) is truly all-or-nothing. If you can measure degree of defect you will have more statistical information and a better model.

$\endgroup$
  • $\begingroup$ You say that my approach to find good parameters is invalid...can you provide me with resource were i can check if my whole logistic approach is scientifically correct? $\endgroup$ – Frank Dec 5 '13 at 9:05
  • 1
    $\begingroup$ Your approach is not generally in line with good statistical practice, so I suggest spending some time studying before proceeding. $\endgroup$ – Frank Harrell Dec 5 '13 at 15:50
2
$\begingroup$

Lots could be said here. gung had the best most concise answer. But to add...
(1) If the cutoff I use is 50%, a max of 25% of the defects get detected is this how you want to define the quality of your model? You can detect all the defects if you change your cutoff.
(2) split sample is suboptimal.
(3) prediction is hard
(4) classically the first approach is to see if your model is miss-specified, could be better specified. Should a variable be categorial rather than linear. Is the relationship between the variable and outcome non-linear. Should you consider interaction terms. But none of these will magically make a poorly predicting model suddenly great.

$\endgroup$
-1
$\begingroup$

There is no direct answer for this question, you have to try different things. From my experience, you can try different methods to boost your sensitivity:

1- If you have a lot of missing values you could try imputed methods such as KNN.

2- Try different Machine learning algorithms SUCH AS svm AND rf RATHER THAN LOGISTIC REGRESSION

3- If you have large number of features so I suggest you to try different features selection such as LASSO

4- Be careful to tune your parameters

5- If none of the above works, you can try Ensemble methods such as bagging, boosting and stacking. Integrating different models will give you a good boost.

Good luck

$\endgroup$
  • 2
    $\begingroup$ Sorry, this is not a good answer. You don't justify any of these methods, or give any kind of prioritization for when and how they will work. Put yourself in the place of a new practitioner, should I try all these things? Are they always relevant? Is there anything the OP can improve about their original procedure? $\endgroup$ – Matthew Drury Mar 20 '18 at 0:47
  • $\begingroup$ I am trying to help, why you loss me 2 points. $\endgroup$ – forever Mar 20 '18 at 1:34
  • 1
    $\begingroup$ So that you are motivated to improve your answer. Thats how the voting system is designed to work. If you adress some of the issues i mentioned above, i will upvote. $\endgroup$ – Matthew Drury Mar 20 '18 at 1:52
  • $\begingroup$ I update my answer, could I get your feedback? $\endgroup$ – forever Mar 24 '18 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.