0
$\begingroup$

I am working on a healthcare data set for breast cancer patients. This data set is class imbalances and the distribution of positive and negative classes is 80%/20%. In order to deal with the class imbalance problem, I am trying to use several outside sampling techniques including down-sampling, up-sampling & SMOTE and comparing the ROC values on training and test set. The sampling is only used on the training set and the test set is left as it is. the first 100 rows of the data are given below

dput(data[1:100,])

structure(list(lvvalue = c(65, 55, 69, 71, 60, 66, 50, 65, 72, 
80, 61, 56, 69, 62, 70, 57, 60, 88, 60, 58, 55.7, 74, 63, 69, 
65, 57, 65, 60, 56.4, 71, 60, 59, 62, 50, 59, 54.1, 73, 69, 61.9, 
51, 56, 70, 72, 81, 91, 78.8, 80, 64, 61, 55, 65, 70, 66, 65, 
66, 62.6, 56, 60, 82, 78, 76, 72, 63, 69, 61, 65, 74, 60, 65, 
55, 65.5, 74, 84, 62.9, 62, 75, 59, 61, 72, 52, 69, 65, 66, 70, 
70, 72, 69, 63, 75, 67, 69, 69, 58, 68, 72, 68, 64, 63, 75, 56
), bilirubin = c(0.4, 11, 7, 8, 0.6, 0.3, 0.92, 12, 5.8, 0.4, 
3, 4.5, 8.5, 0.7, 6.2, 7, 5, 4.2, 0.5, 0.3, 11.7, 0.3, 0.25, 
10, 0.58, 10, 0.2, 0.24, 0.7, 9, 0.5, 8, 12, 0.09, 8, 14.3, 11.7, 
0.5, 8.6, 6, 0.31, 0.9, 0.8, 8, 5, 0.47, 1.22, 0.2, 0.15, 0.7, 
5, 5, 4, 20, 0.6, 3.2, 0.7, 14.7, 0.5, 0.4, 0.6, 0.4, 0.8, 7, 
11.7, 4.3, 12, 0.57, 0.52, 0.5, 0.6, 7.2, 6.4, 0.1, 0.5, 0.7, 
8.55, 0.84, 0.4, 0.3, 0.3, 0.8, 10, 0.2, 0.3, 6.6, 6.6, 0.44, 
7, 0.4, 0.2, 3.8, 0.35, 6, 0.4, 0.9, 7, 3, 10, 0.87), alat = c(26, 
34, 12, 14, 15, 7, 12, 24, 33, 15, 24, 0.37, 31, 16, 13.4, 23, 
5, 40, 7, 21, 0.34, 10, 15, 28, 6, 0.18, 10, 18, 17, 13, 48, 
0.33, 16, 35, 7, 0.37, 38, 10, 20.1, 3, 19, 18, 63, 43, 22, 43.5, 
1.16, 36, 6, 12, 10, 13, 27, 20, 47, 4, 16, 32, 26, 19, 18, 29, 
12.2, 27, 12, 14, 20, 9, 15, 30, 10, 15, 54, 46, 10, 31, 9, 42, 
13, 20, 15, 24, 23, 35, 3, 32, 27, 19, 24, 11, 25, 32, 9, 12, 
9, 16, 22, 11, 40, 30), asat = c(7, 18, 23, 8, 27, 15, 19, 27, 
22, 14, 33, 0.36, 26, 21, 18.4, 23, 16, 21, 14, 23, 0.24, 11, 
13, 22, 7, 0.18, 9, 37, 16, 21, 37, 0.22, 20, 20, 15, 0.57, 23, 
15, 15.3, 9, 20, 16, 56, 28, 18, 17.6, 15, 32, 10, 16, 12, 16, 
23, 20, 44, 8, 31, 15, 13, 13, 15, 22, 13.7, 27, 16, 19, 16, 
17, 9, 24, 16, 17, 18, 46, 14, 29, 12, 29, 18, 21, 13, 27, 21, 
22, 7, 11, 17, 16, 18, 16, 14, 15, 14, 16, 10, 19, 23, 8, 21, 
30), alkaline_phosphatase = c(61, 58, 60, 58, 78, 51, 99, 98, 
77, 54, 55, 1.32, 72, 62, 217.5, 148, 80, 104, 64, 98, 2.2, 77, 
85, 96, 104, 1.01, 87, 72, 82, 262, 95, 1.09, 66, 81, 50, 111, 
76, 102, 126.6, 32, 83, 43, 76, 92, 61, 58, 290, 119, 158, 85, 
53, 47, 81, 77, 66, 116, 64, 79, 51, 56, 88, 45, 76, 78, 176, 
53, 53, 71, 115, 69, 187, 71, 111, 138, 70, 70, 37, 111, 77, 
45, 67, 87, 78, 43, 86, 76, 66, 149, 48, 52, 125, 75, 151, 56, 
132, 60, 94, 41, 89, 96), creatinine = c(0.8, 52, 0.06, 50, 0.9, 
0.8, 0.72, 48, 0.92, 0.8, 0.09, 81, 0.82, 0.7, 107.2, 63, 52, 
0.61, 0.9, 1.1, 66, 0.9, 0.6, 78, 0.63, 59, 0.8, 1, 0.89, 87, 
0.7, 93, 76, 1, 68, 65.7, 0.87, 0.92, 85.5, 68, 0.71, 0.7, 0.7, 
77, 62, 0.54, 0.69, 0.7, 0.98, 0.8, 67, 80, 86, 65, 1.2, 6.4, 
0.7, 0.82, 0.8, 0.9, 0.9, 1.1, 0.6, 60, 60, 4.9, 63, 0.7, 0.78, 
0.7, 0.78, 73, 0.68, 0.7, 0.92, 0.8, 78.7, 0.74, 0.7, 0.8, 0.9, 
0.7, 0.03, 0.7, 0.8, 0.76, 0.81, 0.7, 0.08, 0.7, 0.7, 0.82, 0.83, 
69, 0.68, 0.7, 81, 68, 98, 1), age = c(43L, 39L, 53L, 48L, 66L, 
40L, 51L, 58L, 64L, 44L, 60L, 61L, 53L, 64L, 48L, 57L, 44L, 61L, 
61L, 57L, 54L, 39L, 61L, 53L, 50L, 65L, 70L, 49L, 54L, 55L, 58L, 
60L, 36L, 57L, 52L, 50L, 56L, 34L, 31L, 48L, 53L, 48L, 46L, 50L, 
59L, 65L, 58L, 64L, 56L, 40L, 43L, 50L, 59L, 58L, 46L, 50L, 70L, 
48L, 53L, 43L, 59L, 65L, 45L, 49L, 69L, 50L, 65L, 53L, 57L, 65L, 
57L, 46L, 54L, 48L, 52L, 64L, 50L, 55L, 36L, 37L, 60L, 49L, 67L, 
57L, 35L, 52L, 48L, 68L, 61L, 44L, 45L, 45L, 38L, 32L, 67L, 50L, 
58L, 55L, 60L, 59L), resaln = c(21L, 25L, 29L, 19L, 16L, 24L, 
14L, 27L, 26L, 8L, 11L, 10L, 15L, 11L, 10L, 8L, 8L, 8L, 19L, 
11L, 12L, 24L, 8L, 14L, 12L, 16L, 33L, 6L, 7L, 14L, 20L, 14L, 
26L, 17L, 19L, 8L, 14L, 9L, 26L, 19L, 11L, 7L, 20L, 14L, 8L, 
25L, 10L, 10L, 16L, 11L, 11L, 11L, 23L, 10L, 9L, 18L, 10L, 20L, 
13L, 16L, 13L, 14L, 14L, 15L, 23L, 22L, 7L, 14L, 20L, 20L, 14L, 
6L, 26L, 23L, 12L, 12L, 22L, 14L, 11L, 15L, 15L, 14L, 11L, 15L, 
15L, 12L, 11L, 23L, 13L, 5L, 17L, 16L, 21L, 10L, 15L, 8L, 9L, 
19L, 13L, 20L), posaln = c(3L, 18L, 28L, 8L, 2L, 5L, 2L, 1L, 
8L, 2L, 1L, 1L, 2L, 2L, 1L, 5L, 8L, 1L, 6L, 1L, 2L, 1L, 3L, 11L, 
2L, 1L, 2L, 1L, 3L, 10L, 20L, 3L, 12L, 11L, 11L, 1L, 6L, 2L, 
15L, 4L, 4L, 3L, 2L, 1L, 3L, 5L, 1L, 4L, 11L, 3L, 1L, 1L, 1L, 
1L, 1L, 7L, 6L, 1L, 6L, 1L, 2L, 1L, 10L, 10L, 23L, 2L, 2L, 2L, 
4L, 3L, 1L, 2L, 24L, 4L, 4L, 3L, 7L, 1L, 1L, 4L, 2L, 2L, 4L, 
4L, 1L, 3L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 4L, 2L, 3L, 2L, 1L, 1L, 
11L), histype_idc = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
0L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 
1L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 0L, 1L, 0L, 0L, 
0L, 0L, 0L, 1L, 1L, 1L, 0L, 1L, 1L, 1L), histype_ilc = c(0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 
0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 
0L, 0L, 0L), histype_other = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
1L, 1L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L), ncgr_cba = c(0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L), ncgr_md = c(0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 
0L, 1L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 
1L, 1L, 0L, 1L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 
1L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 
1L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 1L, 
0L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L), ncgr_pd = c(1L, 
1L, 1L, 1L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L, 
0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 
1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 
1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
1L, 0L, 0L), ncgr_wd = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L), ptsize = c(3, 4, 
2.7, 2, 0.9, 3, 3, 0.9, 3, 4.5, 1.7, 2, 6.2, 1.2, 2.5, 1.6, 2.2, 
3, 1.6, 3.6, 2, 2, 2.5, 5, 3, 1.8, 3.5, 1.6, 2.7, 2.7, 3, 1.5, 
4, 5, 2, 2, 1.5, 3.5, 5, 2.7, 1.2, 2.1, 4.5, 1.1, 2.5, 1.4, 1.8, 
2.3, 1.3, 1.5, 2.5, 8, 1.8, 2.1, 3.5, 3.5, 3, 1.5, 2.5, 2.1, 
1.8, 2, 2.4, 3.5, 3.8, 2, 1, 2.5, 1.6, 1.6, 4.5, 2, 3, 4, 1.5, 
1.7, 4, 0.7, 2.3, 1.8, 4, 1.7, 7, 2.8, 1.4, 0.8, 2.5, 0.9, 1, 
2.5, 5, 3.2, 2, 2, 2.2, 2, 3, 2.5, 1.5, 6.5), tnm_val = c(54L, 
54L, 54L, 53L, 53L, 54L, 54L, 53L, 54L, 54L, 53L, 53L, 55L, 53L, 
54L, 53L, 54L, 54L, 53L, 54L, 53L, 53L, 54L, 54L, 54L, 53L, 54L, 
53L, 54L, 54L, 54L, 53L, 54L, 54L, 53L, 53L, 53L, 54L, 54L, 54L, 
53L, 54L, 54L, 53L, 54L, 53L, 53L, 54L, 53L, 53L, 54L, 55L, 53L, 
54L, 54L, 54L, 54L, 53L, 54L, 54L, 53L, 53L, 54L, 54L, 54L, 53L, 
53L, 54L, 53L, 53L, 54L, 53L, 54L, 54L, 53L, 53L, 54L, 53L, 54L, 
53L, 54L, 53L, 55L, 54L, 53L, 53L, 54L, 53L, 53L, 54L, 54L, 54L, 
53L, 53L, 54L, 53L, 54L, 54L, 53L, 55L), ptsite_Left = c(0L, 
0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 1L, 1L, 
1L, 0L, 0L, 1L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 
1L, 0L, 1L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 0L, 1L, 0L, 
1L, 0L, 1L, 0L, 1L, 1L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 1L, 1L, 
0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 1L, 0L, 0L, 1L, 1L, 0L, 1L, 0L, 
0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 1L, 
1L, 0L, 1L), ptsite_Right = c(1L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 
1L, 1L, 1L, 0L, 1L, 1L, 1L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 1L, 
0L, 1L, 1L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 1L, 0L, 1L, 0L, 1L, 0L, 
1L, 1L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 1L, 0L, 1L, 0L, 0L, 1L, 
1L, 1L, 0L, 1L, 1L, 0L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 
1L, 0L, 1L, 1L, 0L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 
1L, 1L, 1L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 0L), evidis_no = c(NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, 1L, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 1L, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, 1L, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA), evidis_yes = c(NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 0L, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, 0L, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 0L, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA), wbc = c(8.6, 7.2, 
5, 4.98, 9.67, 7.26, 4.9, 11.3, 7.3, 11.3, 9.5, 6.2, 6.47, 8.8, 
8.43, 6.5, 6.5, 7.6, 9.2, 6.9, 6.4, 8.5, 5.77, 6.3, 8.1, 4.9, 
5.94, 6.1, 5.97, 4.53, 5, 6.92, 10.1, 8.4, 6.18, 5.5, 5.2, 5.6, 
6.87, 5.7, 7.9, 8.68, 9.9, 5.2, 8.1, 7.92, 6.81, 9, 9.3, 6.6, 
6.9, 4.2, 9.89, 8.59, 5.6, 7.3, 7.7, 9.71, 6.32, 7.3, 4.8, 9.5, 
4.8, 4.7, 6.5, 6.71, 5.7, 6.5, 9.65, 11.4, 5.89, 5.51, 3.8, 7.6, 
7.46, 6.9, 8.1, 7, 8.2, 5.5, 6.3, 5.5, 9.29, 7, 6.6, 5.1, 8.9, 
5.45, 10.4, 11, 9.15, 3.7, 6, 12, 9.55, 6.8, 5.7, 9.6, 7.6, 7.7
), platelets = c(340, 261, 290, 284, 210, 211, 149, 305, 261, 
391, 207, 345, 400, 339, 235, 325, 203, 232, 254, 250, 260, 434, 
502, 305, 289, 208, 290, 232, 238, 301, 308, 277, 260, 350, 323, 
215, 214, 188, 265, 225, 298, 210, 466, 213, 244, 285, 281.1, 
436, 350, 313, 282, 238, 346, 465, 263, 270, 265, 420, 354, 377, 
259, 281, 341, 362, 332, 183, 187, 259, 439, 403, 261, 281, 428, 
450, 251, 280, 308, 321, 306, 188, 231, 236.5, 259, 192, 350, 
240, 367, 299, 301, 264, 347, 250, 264, 456, 238, 225, 316, 288, 
240, 247), hemoglobin = c(13.7, 12.6, 13.6, 10, 13.1, 12.5, 12.88, 
14, 13.3, 12.5, 14.6, 11.9, 12.2, 14.2, 14.3, 13.3, 12.2, 12.8, 
14.7, 12.9, 13.6, 10.8, 13.2, 12.7, 14.5, 12.4, 13.6, 13.3, 14.3, 
12.2, 13.9, 13.3, 10.2, 13.5, 11, 12.9, 12.5, 14.7798, 12.9, 
14.2, 14.168, 14.4, 13.1, 13.1, 11.2, 14.5, 14.1, 10.7, 15.2, 
12.2, 11.3, 10.7, 14, 13.9, 14.6, 10.35, 14.8, 10, 11.6, 11, 
13.7, 11.9, 12.6, 13.3, 12.8, 15.3, 12.7, 13.8, 14.4, 12.4, 13.3, 
12.3, 10.5, 12.4, 14.4, 12.9, 13.1, 14.49, 12.7, 12.8, 11.7, 
14.7, 15.2, 12.2, 12.9, 12.7, 12.2, 11.7, 13.9, 12.6, 12.7, 12.7, 
12.5, 14.4, 12.7, 12.5, 11.9, 16.1, 13.3, 14.6), neutrophils = c(4.988, 
4.76, 2.1, 3.3, 6.2, 4.52298, 2.7, 8.3, 5.1, 8.4, 4.7, 3.4, 4.03, 
4.7, 5.77, 4.3, 4.5, 4.7, 6.532, 4.347, 2.67, 5.95, 3.75, 3.7, 
5.5, 3.5, 3.8, 3.5, 2.77008, 2.8, 2.95, 4.27656, 6.1, 6, 3.708, 
2.915, 2.8, 3.7, 3.95, 3.4, 5.1, 6.4, 6.435, 2.95, 5.1, 5.29056, 
3.657, 4.14, 4.7, 3.7, 3.5, 2.36, 6.95, 5.05, 2.794, 4.3, 4.9, 
6.2, 3.7288, 3.212, 2.784, 6.1, NA, 2.7, 4.6, 4.899, 3.79, 4.3, 
5.62, 9.006, 2.86, 3.43, 2.6, 4.864, 4.74, 4.5, 6.237, 4.2, 6.1, 
2.8, 3.1, 3.3, 6.36, 4.5, 2.772, 3.3, 5.4, 3.07, 6.66, 8, 6.3, 
2, 31, 7.7, 6.876, 4.4, 2.4, 6.3, 3.53, 4.38), penabnAbnormal = c(0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 
0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 
0L, 0L, 0L), penabnNormal = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
0L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 0L, 1L, 0L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L), ht = c(NA, 147, 
164, 157, 175.26, 161.1, 165, 150.5, 159, 156.21, 157, 158, 159, 
154.94, 158, 154, 168, 156, 160.782, 153.67, 163, 175, 150, 148, 
165, 164, 168, 165, 163, 167, 161.29, 163, 155, 162.56, 151, 
151, 161, 170, 164, 162.56, 155, 167.64, 168, 164.5, 142, 164, 
172, 160.02, 157, 160.02, 168.5, 163, 162.56, 157, 160.02, 173, 
149.86, 168, 160, 160.02, 162.56, 165.5, 159, 160, 151, 170, 
160, 171.45, 164, 151.13, 170, 170.5, 164, 162.052, 155, 170.18, 
162, 164, 159.258, 171.45, 160.02, 167.64, 168.5, 154.94, 170, 
158, 155, 161, 158, 162.56, 165, 167, 166, 158, 167, 171, 166, 
158.75, 165, 155), wt = c(NA, 47.1, 61.6, 51, 80.5, 52.6, 54, 
49, 76, 21.36, 78, 60, 87, 27.94, 57, 68, 65.6, 71, 23.13, 36.51, 
80, 70, 51.3, 52.9, 50, 52, 75, 52, 65, 65, 23.86, 78, 56.2, 
24.49, 57, 56.2, 66.5, 58, 62, 22.23, 64, 34.47, 63, 87.6, 70, 
71, 70, 35.38, 75, 27.22, 70.2, 77.1, 39.46, 79.7, 25.86, 101, 
30.84, 68, 57.2, 29.76, 32.21, 70.3, 73, 55, 74.5, 53, 85, 36.29, 
64, 31.75, 70, 65.8, 70, 23.27, 63.9, 24, 64, 67, 22.68, 23.59, 
34.02, 23.59, 71, 29.48, 58, 61, 52, 52, 69, 29.48, 75, 69, 52.3, 
53, 81, 66, 76, 35.38, 100, 66), eintna_Non_significant_abnormalities = c(0L, 
0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 
1L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 
0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 0L, 1L, 
1L, 1L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 
1L, 0L, 0L), eintna_Within_Normal_Limits = c(1L, 1L, 1L, 1L, 
0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 0L, 1L, 1L, 
1L, 1L, 0L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 0L, 0L, 0L, 1L, 
1L, 1L, 1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 1L
), ersta = c(1L, 1L, 1L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 
2L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 1L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 1L, 2L, 2L, 
2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), pgrsta = c(1L, 1L, 1L, 2L, 1L, 
1L, 1L, NA, 1L, 2L, 1L, 2L, 2L, 1L, 1L, 2L, 2L, 1L, 2L, 1L, 2L, 
2L, 1L, 1L, 1L, 2L, 1L, 1L, 2L, 2L, 2L, 2L, NA, 2L, 2L, 1L, 2L, 
2L, 2L, 2L, 2L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
NA, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 1L, 2L, 1L, 1L, 2L, 
2L, 2L, 2L, 1L, 2L, 2L, 1L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 1L, 2L, 2L, 2L, 2L, 1L), 
    class = c(1L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 
    1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 1L, 0L, 
    1L, 0L, 1L, 1L, 1L, 0L, 0L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 
    1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 0L, 0L, 
    1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 
    0L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
    1L, 1L, 0L, 1L, 1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L)), .Names = c("lvvalue", 
"bilirubin", "alat", "asat", "alkaline_phosphatase", "creatinine", 
"age", "resaln", "posaln", "histype_idc", "histype_ilc", "histype_other", 
"ncgr_cba", "ncgr_md", "ncgr_pd", "ncgr_wd", "ptsize", "tnm_val", 
"ptsite_Left", "ptsite_Right", "evidis_no", "evidis_yes", "wbc", 
"platelets", "hemoglobin", "neutrophils", "penabnAbnormal", "penabnNormal", 
"ht", "wt", "eintna_Non_significant_abnormalities", "eintna_Within_Normal_Limits", 
"ersta", "pgrsta", "class"), row.names = c(NA, 100L), class = "data.frame")

The trainControl object is using "repeatedcv" repated 10 times and the metric to optimize is ROC/AUC. "Train" function from the caret package is used with "gbm". The values along with summary statistics are given below for training and test set.

Training Set Performance

Models: original, down, up, smote 
Number of resamples: 100 

ROC 
           Min. 1st Qu. Median   Mean 3rd Qu.   Max. NA's
original 0.5497  0.6588 0.7105 0.7006  0.7412 0.8525    0
down     0.4649  0.6526 0.6901 0.6942  0.7544 0.9561    0
up       0.8210  0.8698 0.8919 0.8892  0.9090 0.9542    0
smote    0.8457  0.8929 0.9091 0.9096  0.9247 0.9609    0

Test Set Performance

            lower       ROC     upper
original 0.4481157 0.5018884 0.5556612
down     0.3935113 0.4471393 0.5007673
up       0.4408710 0.4976186 0.5543662
smote    0.4449739 0.4961928 0.5474117

You can clearly see that the performance for test set decreases drastically leaving me asking the question if I am doing something wrong. I will appreciate some insights onto why this is happening and if there is something I am missing.

Thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In my experience, this is just how gbm works. I never trust the performance on the training sample and always use a test sample to measure how well the model will generalize. Coming from a typical regression background we are taught to be very wary of performance that degrades so much from train to test. But rather than question the model, I think it requires a shift in thinking.

Take, for instance, the randomForest model that builds decision trees with only 1 observation in every terminal node. This model produces perfect predictions on the training data which drops precipitously on a hold out. Yet it is still regarded as one of the best off-the-shelf algorithms around.

I think GBM falls somewhere in-between a memorizing algorithm like RF or KNN and a heavily-biased technique like regression. My advice is to only care about the test-set performance.

Edit

You can also use the k-fold pattern and reserve the out-of-fold estimates. This is as close as you can get to measuring unbiased model performance on actual development records.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.