CV score vastly different from Train-Test score

Question

I'm working on a multi-class classification task. I'm currently trying to tune a LGB model but have encountered a behavior that I do not understand. First, my data is from 1996 to 2015 so I split my data into a training and validation set like this:

YEAR_START = 1996
YEAR_TRAIN = 2010
YEAR_VAL = 2015

X = df_train.drop(columns='Label')
y = df_train['Label'].values

X_train = df_train[(df_train['year'] >= YEAR_START) & (df_train['year'] < YEAR_TRAIN)].drop(columns='Label')
y_train = df_train[(df_train['year'] >= YEAR_START) & (df_train['year'] < YEAR_TRAIN)]['Label'].values

X_val = df_train[(df_train['year'] >= YEAR_TRAIN) & (df_train['year'] <= YEAR_VAL)].drop(columns='Label')
y_val = df_train[(df_train['year'] >= YEAR_TRAIN) & (df_train['year'] <= YEAR_VAL)]['Label'].values

I've checked that X_train and X_val have significant data (30k and 10k). Then, I train my LGB Model like this:

paramsLGB = {
'objective': 'multiclass',
'num_class': 3,
'boosting_type': 'gbdt',
'learning_rate': 0.05,
'num_leaves': 31,
'max_depth': -1,
'colsample_bytree': 0.8,
'min_child_weight': 1,
'min_child_samples': 20,
'subsample': 0.8,
'subsample_freq': 1,
'reg_lambda': 0.0,
'reg_alpha': 0.0,
'min_split_gain': 0.0,
'force_col_wise': True,
'verbose': 0
}

Then, my cross-validation (notice the folds are only taken from X_train):

num_folds = 5
kf = KFold(n_splits=num_folds, shuffle=False)

results = []

for train_idx, val_idx in kf.split(X, y):
   X_trainCV, X_valCV = X.iloc[train_idx], X.iloc[val_idx]
   y_trainCV, y_valCV = y[train_idx], y[val_idx]

   d_train = lgb.Dataset(X_trainCV, label=y_trainCV)
   d_val = lgb.Dataset(X_valCV, label=y_valCV, reference=d_train)

# Train the model with CV
num_round = 1000  # Number of boosting rounds
callbacks = [lgb.early_stopping(50), lgb.log_evaluation(period=10)]

clf = lgb.train(paramsLGB, d_train, num_round, valid_sets=[d_train, d_val], 
callbacks=callbacks)

# Calculate accuracy on the validation set and print it
y_pred = clf.predict(X_val, num_iteration=clf.best_iteration)
y_pred_class = [list(x).index(max(x)) for x in y_pred]
accuracy = accuracy_score(y_val, y_pred_class)

print(f'Accuracy on validation set: {accuracy}')

Which outputs this (and similar for other folds):

Early stopping, best iteration is:
[54]    training's multi_logloss: 0.238541  valid_1's multi_logloss: 0.382562
Accuracy on validation set: 0.8845628415300546

Great! 88% accuracy!

But, when I run this afterwards:

m = lgb.LGBMClassifier(**paramsLGB, random_state=51)

m.fit(X_train, y_train)
acc = accuracy_score((y_val), (m.predict(X_val)))
print("The classification accuracy on test set of the LGBM: {:.4f}".format(acc))

I get a very different accuracy:

The classification accuracy on test set of the LGBM: 0.7981

Could anyone explain what's going on here? I'm not sure if it's a bug in my code or in my machine learning methodology. I could see a small drop in accuracy, but a 10% drop seems weird to me. This might (?) have something to do with the fact that classes are kind of imbalanced (80-15-5). Would I need to handle class imbalance here?

Answer 1

The biggest problem that you seem to have is the split in years. The train range years is 1996 - 2010, and the test set range years is 2010-2015. In the test you use years 1996-2010 to predict years 2010-2015. However, in the cross validation, you random sample observations from the train set, mixing the years, making the problem easier (if there is a change throughout time).

You can avoid this problem in the cross validation by taking sequentially the years. I.e., use 1996 - 2000 to predict 2000-2005, 1996 - 2001 to predict 2001 - 2006 and so on. This will be closer for the task your evaluating with train-test.