3
$\begingroup$

Okay so I run this model manually and get around 80-90% accuracy:

mlp = MLPClassifier(hidden_layer_sizes=(
    50, 50), activation="logistic", max_iter=500)
mlp.out_activation_ = "logistic"
mlp.fit(X_train, Y_train)
predictions = mlp.predict(X_test)
print(confusion_matrix(Y_test, predictions))
print(classification_report(Y_test, predictions))

Then, I do some 10-fold cross validation:

print(cross_val_score(mlp, X_test, Y_test, scoring='accuracy', cv=10))

And I get accuracy stats something like the following for each fold:

[0.72527473 0.72222222 0.73333333 0.65555556 0.68888889 0.70786517 0.69662921 0.75280899 0.68539326 0.74157303]

I've done this about 5 times now. Every time I run the model on its own, I get 80-90% accuracy, but then when I run cross-validation, my model is averaging 10-20% less than when the model is run once manually.

The chances of getting the best model first time, five times in a row are 1 in 161,051 (1/11 ^ 5). So I must just be doing something wrong somewhere.

Why does my model consistently perform worse in cross-validation?

EDIT - I'd like to add that I'm doing exactly the same thing with a RandomForestClassifier() and getting expected results, i.e. the accuracy obtained when I run the model manually is around the same as when run by the cross_val_score() function. So what is it about my MLPClassifier() that's producing this mismatch in accuracy?

$\endgroup$
5
  • 2
    $\begingroup$ I don't get what you mean by your odds being of 'randomly getting the best model the first time'. You're optimizing. Against your training set. There's nothing random about it. $\endgroup$
    – jkm
    Dec 10, 2019 at 21:20
  • $\begingroup$ As in, I run the model on its own. I get a high accuracy. I don't change the hyper-parameters. Then I run cross validation. Each fold is getting worse accuracy than when I ran the model manually the first time. Randomly was probably the wrong word, I just mean that the chances of my getting the best model first time, five times in a row are small. Apologies if the question is porrly worded. $\endgroup$ Dec 10, 2019 at 21:23
  • $\begingroup$ My question is why is accuracy worse, when the model is run by the cross_val_score() function, in contrast to when being run manually by me? $\endgroup$ Dec 10, 2019 at 21:26
  • $\begingroup$ Are you running the 10-fold CV on the train-and-test sets concatenated together? Or using your training set from the train-test split to both train and test yout model during cross validation? $\endgroup$ Dec 10, 2019 at 22:11
  • $\begingroup$ Hi, how does your MLP classifier do on the train set in holdout-validation and what split do you use? Moreover, what accuracy scores (train and test for holdout as well as CV) does your RandomForestClassifier() have? $\endgroup$
    – Jonathan
    Dec 10, 2019 at 22:41

2 Answers 2

1
$\begingroup$

I think there is some confusion as to the basis of what is being observed here. First, a model is trained against the X_train/Y_train dataset. When testing this model against the X_test/Y_test (holdout) dataset, an accuracy of 80-90% is observed. Next, a cross-validation was run. This outputs a fold score based on the X_train/Y_train dataset.

The question asked was why the score of the holdout X_test/Y_test is different than the 10-fold scores of the training set X_train/Y_train. I believe the issue is that based on the code given in the question, the metrics are being obtained on different datasets. The 80-90% score comes from running mlp.predict() against the test dataset, while the 60-70% accuracy comes from obtaining fold scores for the train dataset.

$\endgroup$
6
  • $\begingroup$ But the the cross_val_score() function is taking X_train and Y_train. Not X_test and Y_test. $\endgroup$ Dec 10, 2019 at 21:41
  • $\begingroup$ Alright I've changed it so that the cross_validation_score() now takes X_test and Y_test, like predictions. There is still a mismatch. Editing the question now. $\endgroup$ Dec 10, 2019 at 21:47
  • $\begingroup$ I reworded my answer for clarity. What you should try is mlp.predict(X_train) to see if perhaps the train set is simply lower scoring than the test set. Because the score from cross_val_score() is pulled from the train set, not test set unless I am still misunderstanding what was done. $\endgroup$ Dec 10, 2019 at 21:51
  • $\begingroup$ But if the model has been trained on X_train, surely the accuracy of testing on X_train should be higher than X_test, not lower. Because X_test is unseen data. $\endgroup$ Dec 10, 2019 at 21:52
  • $\begingroup$ That's usually, but not always, the case. Sometimes the train set does poor because it is larger, or has more outliers, or just out of statistical bad luck. I may be wrong, of course, not having access to your data. $\endgroup$ Dec 10, 2019 at 21:54
3
$\begingroup$

Context: The cross-validation method and the holdout method (train-test split) are seen as two methods to evaluate the model performance. The goal of this evaluation is to obtain an estimate of the generalization (or, test) error.

Summary: If the accuracy from the cross-validation method is less than the accuracy from the holdout method, it indicates model overfitting.

Explanation: When the test error is estimated by the holdout method, the data is split into the training and holdout samples. However, this split may induce a bias since there's no guarantee of a randomness within the training and test samples even if the whole dataset is considered a random sample. In order to mitigate this bias we can average the test error stemming from different test samples. This is precisely what cross-validation does - it rotates the test sample across the whole dataset and for every test sample, the remaining dataset becomes the training sample. For each split, the test error is computed after fitting the model over the corresponding training sample. The test errors from each split are averaged to obtain the average test error, or the cross-validated error.

In the absence of cross-validation, it is possible that the model becomes biased by the (biased) data split. This results in overfitting. Overfitting is the result of the model memorizing the training examples (and thus capturing noise) than actually learn (or identify the true pattern/relationship) from the training examples.

Only when there is no noise in the data (which is unlikely in the real world) and assumed model reflects the true relationship (which is typically difficult to know without domain knowledge), the holdout and cross-validation methods provide the same accuracy.

Hope this helps !

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.