I'm trying to understand why my train AUC = 1 while my test AUC is near 58% using random forest.
- Context: You are trying to sell a product, and you have historic data about the purchases/noPurchases of such product, from March 2020 to August 2020. You leave August for test data, and the rest for training data. It results approximately in 80% train and 20% test. Each purchase/noPurchase is related to one client, and the data available for that client is the month before the purchase. For example: If a client purchase/noPurchase on July, the data available for that client in that month is June.
- Data (train+test): 75 columns and 10k rows, target variable is binary with 90%-10% imbalance. All data is numeric.
- Modeling: The scoring is ROC_AUC, and all predictions should be probabilities (to plot roc curve)
train = X_train + y_train;
test = X_test + y_test
# STANDARDIZE AND IMPUTE TRAIN AND TEST SEPARATED scaler = StandardScaler() imputer = KNNImputer() X_train_scaled = scaler.fit_transform(X_train) X_train_scaled = imputer.fit_transform(X_train_scaled) X_test_scaled = scaler.fit_transform(X_test) X_test_scaled = imputer.fit_transform(X_test_scaled) # MODEL model = RandomForestClassifier() # we instantiate the model model.fit(X_train_scaled, y_train) # fit y_train_predictions = model.predict_proba(X_train_scaled) # predict # EXTRACT TRAIN CLASSES TO PLOT for i,k in enumerate(model.classes_ == 1): print(i,k) if k == True: y_train_predictions = y_train_predictions[:,i] y_test_predictions = model.predict_proba(X_test_scaled) # predict # EXTRACT TEST CLASSES TO PLOT for i,k in enumerate(model.classes_ == 1): print(i,k) if k == True: y_test_predictions = y_test_predictions[:,i] # PLOT fpr, tpr, _ = roc_curve(y_train, y_train_predictions) plt.plot(fpr, tpr) fpr, tpr, _ = roc_curve(y_test, y_test_predictions) plt.plot(fpr, tpr)
And the problem is that
train AUC = 1 and
test AUC = 58% approx. I thought on these possible causes:
- The dependent variable is included in the
X_trainmatrix. (Checked, it isn't)
- There is some explanatory variable (e.g. credit card charges of the product, if I have credit card charges data) that is included in
However, I discard both possible causes above because I checked meticulously the first one, and the second one is not possible given that the maximum feature importance is
But still, is there any other possible explanation for
train AUC = 1 and
test AUC = 58%, or something I'm missing?
PS: Here is a very similar question, but all the answers are general given that the OP doesn't specify how is building the model (that's why I put all the code)