I'm trying to understand why my train AUC = 1 while my test AUC is near 58% using random forest.

  • Context: You are trying to sell a product, and you have historic data about the purchases/noPurchases of such product, from March 2020 to August 2020. You leave August for test data, and the rest for training data. It results approximately in 80% train and 20% test. Each purchase/noPurchase is related to one client, and the data available for that client is the month before the purchase. For example: If a client purchase/noPurchase on July, the data available for that client in that month is June.
  • Data (train+test): 75 columns and 10k rows, target variable is binary with 90%-10% imbalance. All data is numeric.
  • Modeling: The scoring is ROC_AUC, and all predictions should be probabilities (to plot roc curve)

Code: train = X_train + y_train; test = X_test + y_test

scaler = StandardScaler()
imputer = KNNImputer()
X_train_scaled = scaler.fit_transform(X_train)
X_train_scaled = imputer.fit_transform(X_train_scaled)
X_test_scaled = scaler.fit_transform(X_test)
X_test_scaled = imputer.fit_transform(X_test_scaled)

model = RandomForestClassifier() # we instantiate the model
model.fit(X_train_scaled, y_train) # fit
y_train_predictions = model.predict_proba(X_train_scaled) # predict

for i,k in enumerate(model.classes_ == 1):
    if k == True:
        y_train_predictions = y_train_predictions[:,i]
y_test_predictions = model.predict_proba(X_test_scaled) # predict

for i,k in enumerate(model.classes_ == 1):
    if k == True:
        y_test_predictions = y_test_predictions[:,i]

fpr, tpr, _ = roc_curve(y_train, y_train_predictions)
plt.plot(fpr, tpr)
fpr, tpr, _ = roc_curve(y_test, y_test_predictions)
plt.plot(fpr, tpr)

enter image description here

And the problem is that train AUC = 1 and test AUC = 58% approx. I thought on these possible causes:

  • The dependent variable is included in the X_train matrix. (Checked, it isn't)
  • There is some explanatory variable (e.g. credit card charges of the product, if I have credit card charges data) that is included in X_train

However, I discard both possible causes above because I checked meticulously the first one, and the second one is not possible given that the maximum feature importance is 0.0429

But still, is there any other possible explanation for train AUC = 1 and test AUC = 58%, or something I'm missing?

PS: Here is a very similar question, but all the answers are general given that the OP doesn't specify how is building the model (that's why I put all the code)

  • $\begingroup$ You are saying using two separate scalers and imputers is misleading, but you also say that test data should be scaled/imputed using a separated scaler ("test data should be scaled/imputed using test data scaler"), which is what I just did. On another point, why there is data leakage if you use separated scalers/imputers? $\endgroup$ – Chris Sep 15 '20 at 12:36
  • 1
    $\begingroup$ Sorry, typo! My bad. Using two separate scalers (and imputers) one for the training and one of the test data is misleading. Test data should be scaled/imputed using the train data scaler otherwise there is leakage. Regularise more; for example, increase the minimum number of instances per leaf and/or reduced the maximum tree depth. Leakage will exist because we do not know beforehand anything about the mean or variance of our test data. Think the scenario we have a single test point, what would be the variance then? $\endgroup$ – usεr11852 Sep 15 '20 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.