# Distinguishing between overfitting and wrong model selection

I built a dozen of different models using caret package for classifying customer purchase habits into 5 categories (catA, catB, catC, catD, none) based on 4 numeric predictors (independent variables).

My dataset contains 10,000 customers that have already been categorised:

Category     Predictor1 Predictor2 Predictor3 Predictor4
catA         1.7211     0.6222    -0.0693     2.9370
catA         2.4935    -1.0456    -1.1256     0.2146
catB         0.2413    -0.8871     0.4987    -0.6123
catC        -0.8276     2.6629     2.9298    -0.1048
catC        -0.4402     2.1939    -1.8686    -0.0591
catC         1.6719     2.9085     2.7920     2.3501
catD        -1.6504    -0.1378    -1.2276     2.7824
catD         2.5917    -0.0586    -1.2981     1.0934
catA        -1.2885    -1.8646     1.3895     2.5428
none         0.8258    -0.3217     0.9551     2.6105


My categories are unbalanced:

. catA: 3,100 samples

. catB: 3,000 samples

. catC: 3,200 samples

. catD: 200 samples

. none: 500 samples

Using this dataset, I applied the following method:

1) Split dataset into 80% training and 20% testing sets

2) Train discriminant analysis models using 10-fold cross validation (repeated 3 times)

3) Test the final model on the test set

From there I selected the 3 best discriminant analysis models with the highest F1 score (I read that it is more suitable than the accuracy measure for unbalanced multi-class classifier).

These 3 models show very close accuracy metrics like so:

  Model   Accuracy   Kappa
model_1      0.943   0.891
model_2      0.931   0.887
model_3      0.947   0.893


For the 3 models, looking at the metrics for each category, we can see that 'catD' and 'none' are less accurate than the others:

  Model    Category Sensitivity Specificity Neg_Pred_Value  Precision       F1  Prevalence  Detection_Rate  Detection_Prevalence    Balanced_Accuracy
model_1        catA      0.9606      0.9744         0.9867     0.9261   0.9430      0.2502          0.2404                0.2595               0.9675
model_1        catB      0.9459      0.9745         0.9800     0.9315   0.9386      0.2683          0.2538                0.2725               0.9602
model_1        catC      0.9853      0.9779         0.9950     0.9365   0.9603      0.2481          0.2445                0.2611               0.9816
model_1        catD      0.5833      0.9957         0.9921     0.7241   0.6461      0.0186          0.0108                0.0150               0.7895
model_1        none      0.3396      0.9857         0.9625     0.5806   0.4285      0.0549          0.0186                0.0321               0.6626


My problem is that when I predict categories on a new dataset (for which I do not know the outcomes), the 3 models give me completely different stories.

For example, 'model_1' assigns only 2 categories in total (e.g. 'catA' and 'none'), whereas 'model_2' predicts 3 different ones (e.g. 'catB', 'catC', 'catD') and model_3 find all categories. Also, the numbers of sample assigned to each category are very different between the 3 models.

Questions:

1. Although all these 3 models show good accuracy metrics on the test set, could they be all overfitted and wrong? (And how could I check that?)
2. Is there a way to know which one of the 3 models is the most accurate when classifying on the new unseen dataset?