For logistic model, I divided dataset into two parts training sample (70% of 360 data point (observation)) and test sample (rest 30% of 360) randomly. After that I built logistic model on training sample and area under curve(AUC) is coming around 82% and Hosmer-Lemeshow (HL) test is also throwing the positive result (H0 accepted p-value = .675) with 11 variables out of 67 but same model when I checked on test sample i found AUC around 71% and HL test showing positive result. Could you please tell me what will be reason behind the AUC drop form 82% to 71% ? and what will be the solution for this?
$\begingroup$
$\endgroup$
8
-
6$\begingroup$ It might be overfitting your training data because you have relatively few samples compared to number of variables. Try to add some L2 regularization and use cross validation to find the right amount. $\endgroup$– stmaxCommented Oct 1, 2015 at 6:40
-
3$\begingroup$ An AUC of 71% is pretty good. The reason for the drop is likely due to overfitting. It could also be caused some some unique features of the validation datasets such as outliers. Have you examined the distributions of the covariates between the training and validation datasets as well? It is possible that there are systematic differences between the two datasets by chance. $\endgroup$– StatsStudentCommented Oct 1, 2015 at 7:20
-
$\begingroup$ Just to clarify, I should say that an AUC is pretty good depending on the subject matter. In social sciences an AUC can be pretty good, but in chemistry, physics, etc. that could be fair to poor. $\endgroup$– StatsStudentCommented Oct 1, 2015 at 8:05
-
$\begingroup$ @StatsStudent, I have already taken care of outliers and i did outlier treatments also. For over-fitting problem, i revisited at least 5 times and the present model is looking fine. Could you please suggest me what else can i do for over-fitting problem and to increase the AUC in the test sample? $\endgroup$– user43247Commented Oct 5, 2015 at 9:52
-
$\begingroup$ Here is something you could try. After you have split your data into a training dataset (TD) and a validation dataset (VD), use a bootstrap aggregating procedure or "bragging" procedure for variable selection. Using this method you generate, say 10K replicated datasets by drawing samples with replacement from your TD so that each replicate is the same size as the original TD. Perform an automated variable selection method on each of the replicated dataset, keeping track of which variables are deemed significant in the process. After iterating through the 10K samples, (continued). . . $\endgroup$– StatsStudentCommented Oct 6, 2015 at 2:54
|
Show 3 more comments