I'm trying to understand why my train AUC = 1 while my test AUC is near 58% using random forest.
- Context: You are trying to sell a product, and you have historic data about the purchases/noPurchases of such product, from March 2020 to August 2020. You leave August for test data, and the rest for training data. It results approximately in 80% train and 20% test. Each purchase/noPurchase is related to one client, and the data available for that client is the month before the purchase. For example: If a client purchase/noPurchase on July, the data available for that client in that month is June.
- Data (train+test): 75 columns and 10k rows, target variable is binary with 90%-10% imbalance. All data is numeric.
- Modeling: The scoring is ROC_AUC, and all predictions should be probabilities (to plot roc curve)
Code: train = X_train + y_train
; test = X_test + y_test
# STANDARDIZE AND IMPUTE TRAIN AND TEST SEPARATED
scaler = StandardScaler()
imputer = KNNImputer()
X_train_scaled = scaler.fit_transform(X_train)
X_train_scaled = imputer.fit_transform(X_train_scaled)
X_test_scaled = scaler.fit_transform(X_test)
X_test_scaled = imputer.fit_transform(X_test_scaled)
# MODEL
model = RandomForestClassifier() # we instantiate the model
model.fit(X_train_scaled, y_train) # fit
y_train_predictions = model.predict_proba(X_train_scaled) # predict
# EXTRACT TRAIN CLASSES TO PLOT
for i,k in enumerate(model.classes_ == 1):
print(i,k)
if k == True:
y_train_predictions = y_train_predictions[:,i]
y_test_predictions = model.predict_proba(X_test_scaled) # predict
# EXTRACT TEST CLASSES TO PLOT
for i,k in enumerate(model.classes_ == 1):
print(i,k)
if k == True:
y_test_predictions = y_test_predictions[:,i]
# PLOT
fpr, tpr, _ = roc_curve(y_train, y_train_predictions)
plt.plot(fpr, tpr)
fpr, tpr, _ = roc_curve(y_test, y_test_predictions)
plt.plot(fpr, tpr)
And the problem is that train AUC = 1
and test AUC = 58%
approx. I thought on these possible causes:
- The dependent variable is included in the
X_train
matrix. (Checked, it isn't) - There is some explanatory variable (e.g. credit card charges of the product, if I have credit card charges data) that is included in
X_train
However, I discard both possible causes above because I checked meticulously the first one, and the second one is not possible given that the maximum feature importance is 0.0429
But still, is there any other possible explanation for train AUC = 1
and test AUC = 58%
, or something I'm missing?
PS: Here is a very similar question, but all the answers are general given that the OP doesn't specify how is building the model (that's why I put all the code)