How to validate models beyond checking for overfitting

Question

I have an unusual problem, which is that my model is performing too well and I am struggling to trust it.

The data is a table of "snapshots" about moments in games of chess. For example, a game that lasted 20 minutes might have 4 observations in the data, corresponding to the state of play at 1, 6, 11 and 16 minutes into the game. Features are things like how many moves my player has made, how many pawns he's lost, whether he's lost his queen and so on. The target feature on each observation is whether or not this player eventually won the game, and that's what I want to predict.

I trained a model (a RandomForestClassfier) on a bunch of data. The outcome was a 0.929 Accuracy and 0.983 AUC score; that score is so high that I struggle to trust it, so i want to validate it as securely as possible.

Strangely, the model doesn't seem to be significantly overfitted:

rfc = RandomForestClassifier(n_jobs=6, max_depth=15)

x_train, x_test, y_train, y_test = train_test_split(
    df_full.drop(
        'Status',
        axis=1
    ),
    df_full.Status
)

rfc.fit(x_train, y_train)

print('Train Score: ',
      rfc.score(x_train, y_train),
      'Test Score: ',
      rfc.score(x_test, y_test),
      'Test AUC Score: ',
      roc_auc_score(y_test, rfc.predict_proba(x_test)[:,1])
     )

Train Score:  0.9350048879188833 Test Score:  0.929179472533281 Test AUC Score:  0.9832906048250383

It's performing slightly better on the training set (750k observations) compared to the testing set (250k), but only by about 0.005 accuracy score. I've also performed cross validation using StratifiedKFold(shuffle=True, n_splits=5) and gotten similar scores. Now the character of some features in the dataset are changing over time. For example for newer observations the "GameLength" feature is trending firmly downwards and the player is getting better and better, winning more often:

So I decided to test the possibility that I'm overfitting to the earlier observations some how by splitting a random sample out of the dataframe, then scoring the model against observations from the sample with CreatedDate < 2018-01-01 and against observations from the sample with CreatedDate >= 2018-01-01. This does seem to bear out that hypothesis:

df_s = df.sample(frac=0.25)
df = df.drop(df_s.index)
df_s_1 = df_s.loc[df_s.CreatedDate < '2018-01-01']
df_s_2 = df_s.loc[df_s.CreatedDate >= '2018-01-01']

rfc = RandomForestClassifier(n_jobs=6, max_depth=15)

x_train, x_test, y_train, y_test = train_test_split(
    df.drop(
        'Status',
        axis=1
    ),
    df.Status
)

rfc.fit(x_train, y_train)

print('Full Train Score: ',
      rfc.score(x_train, y_train),
      'Full Test Score: ',
      rfc.score(x_test, y_test),
      'Holdout Pre 2018 Score: ',
      rfc.score(df_s_1.drop('Status', axis=1), df_s_1.Status),
      'Holdout Post 2018 Score: ',
      rfc.score(df_s_2.drop('Status', axis=1), df_s_2.Status)
     )

Full Train Score:  0.9285909152912291
Full Test Score:  0.9207593281991532
Holdout Pre 2018 Score:  0.9304484856471416
Holdout Post 2018 Score:  0.889780437600061

We can see that the Post-2018 Score is significantly lower than the Pre-2018 score. This problem is not specific to the holdout data however. If I compare the prediction accuracy against the main set, they follow broadly equivalent patterns:

Which would seem to imply the model is reasonably well fit and that it simply predicts new observations less well; presumably because they're drowned out by the much larger number of earlier observations and their different characteristics. With that being the case, I retrained the model on just those observations later than 2018-01-01 to the following results:

df3 = df.loc[df.CreatedDate >= '2018-01-01'].drop(['score', 'Correct_Prediction'], axis=1)

x3_train, x3_test, y3_train, y3_test = train_test_split(df3.drop(['CreatedDate', 'Status', 'LifeBin'], axis=1), df3.Status)

rfc3 = RandomForestClassifier(max_depth=15, n_jobs=6)

rfc3.fit(x3_train, y3_train)

print('Train Score: ',
      rfc3.score(x3_train, y3_train),
      'Test Score: ',
      rfc3.score(x3_test, y3_test)
     )

Train Score: 0.9588592284513856 Test Test Score: 0.9382511713179874

There's now much less of a trend, and the model is back to its previous good level of performance. I'm out of ideas to poke holes in that score now, so my question is is there anything else I can do to test the validity of that score?

Answer 1

Could you please explain your model in more detail. The Python code obscures what's happening, since an X matrix and Y matrix tell us little about the actual model. What are your X's for each sample, and what is your Y -- at what point in time?

Is each observation a point in time and you can have multiple observations from a single game? In that case, the observations are not independent. (Not to mention that if you're not careful, you will be using information "from the future" in your model, which will cause it to perform unrealistically well.)

Is your model, in the end, identifying individual players or competitions and therefore simply predicting that good players (or players that play in certain venues) are more likely to win? Machine Learning will pretty much always find a signal: the question is what is the signal?

As an example of what I mean, I once heard a talk by an incredible data scientist who won a competition to predict the severity of cancer for patients based on MRI results. She looked back at the data after the fact and figured out that what her model had really done was to identify which cancer center the patient had gone to for their MRI -- which MRI machine they had used, actually -- and since different cancer centers tend to see patients with different levels of severity, that was what her model was predicting.