0
$\begingroup$

I am working on a dataset that has 300+ predictors and the dependent variables is very imbalanced (99:1). I need to have a prediction accuracy to show to my client.Here is my analytical process.

  1. clean data: remove incomplete columns and rows, then I have 80% of rows remaining and 100+ predictors.
  2. use LASSO: use LASSO with logistic regression to generate the model (by setting up train and testing sets). Then I have problem finding the best cut points. Below is the accuracy stats for the prediction in testing set if I set cut point as 50%:

pred 0 1 0 825 36 1 23 43

The prediction accuracy is too low and I am wondering if it could be improved by choosing different cut points.

Appreciate any helps and suggestions. Thanks.

$\endgroup$
1
$\begingroup$

Not sure accuracy is a useful term here.

  • Note that the ratio of FP to FN is 36/23 with is 1.5. Fairly close to 1.

  • As you decrease the threshold to below 50% you are going to increase your TP at the expense of increasing your FP. The cost ratio of FP/FN will increase.

  • If you increase your threshold to above 50%, your FP will decrease and your cost ratio of FP/FN will decrease to below 1.

The question is how costly are false negative estimates compared to false positive estimates? Once you have a sense of that ratio you can then set you probability cut-point. Often with rare events one is willing to endure a fair number of false positives and the ratio is >1. Usually a guesstimate at this ratio if fairly good starting point, but some domain knowledge is needed.

Or if you don't want to make a decision, a ROC curve is one way to present the information.

$\endgroup$
  • $\begingroup$ thanks for your answer, which is very helpful. Since I don't have much industry experience, I choose to use ROC curve. Inspired by this post stackoverflow.com/questions/16347507/… $\endgroup$ – Z. Zhang Jan 13 '16 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.