Distinguishing between overfitting and wrong model selection

Question

I built a dozen of different models using caret package for classifying customer purchase habits into 5 categories (catA, catB, catC, catD, none) based on 4 numeric predictors (independent variables).

My dataset contains 10,000 customers that have already been categorised:

Category     Predictor1 Predictor2 Predictor3 Predictor4
    catA         1.7211     0.6222    -0.0693     2.9370
    catA         2.4935    -1.0456    -1.1256     0.2146
    catB         0.2413    -0.8871     0.4987    -0.6123
    catC        -0.8276     2.6629     2.9298    -0.1048
    catC        -0.4402     2.1939    -1.8686    -0.0591
    catC         1.6719     2.9085     2.7920     2.3501
    catD        -1.6504    -0.1378    -1.2276     2.7824
    catD         2.5917    -0.0586    -1.2981     1.0934
    catA        -1.2885    -1.8646     1.3895     2.5428
    none         0.8258    -0.3217     0.9551     2.6105

My categories are unbalanced:

. catA: 3,100 samples

. catB: 3,000 samples

. catC: 3,200 samples

. catD: 200 samples

. none: 500 samples

Using this dataset, I applied the following method:

1) Split dataset into 80% training and 20% testing sets

2) Train discriminant analysis models using 10-fold cross validation (repeated 3 times)

3) Test the final model on the test set

From there I selected the 3 best discriminant analysis models with the highest F1 score (I read that it is more suitable than the accuracy measure for unbalanced multi-class classifier).

These 3 models show very close accuracy metrics like so:

  Model   Accuracy   Kappa
model_1      0.943   0.891
model_2      0.931   0.887
model_3      0.947   0.893

For the 3 models, looking at the metrics for each category, we can see that 'catD' and 'none' are less accurate than the others:

  Model    Category Sensitivity Specificity Neg_Pred_Value  Precision       F1  Prevalence  Detection_Rate  Detection_Prevalence    Balanced_Accuracy
model_1        catA      0.9606      0.9744         0.9867     0.9261   0.9430      0.2502          0.2404                0.2595               0.9675
model_1        catB      0.9459      0.9745         0.9800     0.9315   0.9386      0.2683          0.2538                0.2725               0.9602
model_1        catC      0.9853      0.9779         0.9950     0.9365   0.9603      0.2481          0.2445                0.2611               0.9816
model_1        catD      0.5833      0.9957         0.9921     0.7241   0.6461      0.0186          0.0108                0.0150               0.7895
model_1        none      0.3396      0.9857         0.9625     0.5806   0.4285      0.0549          0.0186                0.0321               0.6626

My problem is that when I predict categories on a new dataset (for which I do not know the outcomes), the 3 models give me completely different stories.

For example, 'model_1' assigns only 2 categories in total (e.g. 'catA' and 'none'), whereas 'model_2' predicts 3 different ones (e.g. 'catB', 'catC', 'catD') and model_3 find all categories. Also, the numbers of sample assigned to each category are very different between the 3 models.

Questions:

1. Although all these 3 models show good accuracy metrics on the test set, could they be all overfitted and wrong? (And how could I check that?)
2. Is there a way to know which one of the 3 models is the most accurate when classifying on the new unseen dataset?

Answer 1

1. Since you are doing cross-validation, the purpose of which is to check for overfitting, you can safely assume that the models are not overfitting till the train and validation accuracy is same. Models are not 'right'/'wrong', they have their respective accuracy or scores as you have outlined.
2. The safest way is to use all the 3 models for prediction and take a vote of 3. This is a type of 'ensemble methods' and you may read up on those.