0
$\begingroup$

I classified some medical images. And distribution of the dataset is :

494 Train Anormal

469 Train Normal

37 Test Normal

64 Test Anormal

84 Val Anormal

37 Val Normal

...

My training result is (by ViT):

loss: 0.2714 - accuracy: 0.9102 - val_loss: 0.2624 - val_accuracy: 0.9196

and test result is:

               precision    recall    f1-score   support

           0       0.57      0.60      0.58        47
           1       0.46      0.43      0.44        37

    accuracy                           0.52        84

   macro avg       0.51      0.51      0.51        84

weighted avg       0.52      0.52      0.52        84
  • So my question is that test prediction is not good because of imbalanced data? Or I should figure out something else?

I know 1000 images are not a nice thing in DL but I have to complete this training with them. Also, I implemented data augmentation.

Improve this question
$\endgroup$
8
  • $\begingroup$ A couple possibilities - (1) you have been using validation extensively to optimize hyperparameters, and thus it is effectively just part of your training set; (2) test data just looks different from training/validation $\endgroup$
    – Paul
    Commented Mar 6, 2022 at 14:29
  • $\begingroup$ 1. I didn't understand your first comment, 2. Do you mean the size of the test is different? $\endgroup$
    – Zehra N.
    Commented Mar 6, 2022 at 15:54
  • 2
    $\begingroup$ On what basis did you state "no overfitting" in the title? $\endgroup$
    – Christian Hennig
    Commented Mar 6, 2022 at 21:56
  • 1
    $\begingroup$ The problem is that the training set is approximately balanced, but your test and validation sets have significantly fewer normal than anormal examples, which is why the classifier is presumably over-predicting "normal" in the test set (difficult to tell because of the labelling). Your training set should be representative of the statistical distribution in the test set (and in operation), and that includes the relative frequencies of the classes. You might want to use stratified resampling to form the training, test and validation sets, so they all have the same label distributions. $\endgroup$
    – Dikran Marsupial
    Commented Mar 6, 2022 at 23:52
  • 1
    $\begingroup$ @DikranMarsupial thank you so much!! $\endgroup$
    – Zehra N.
    Commented Mar 7, 2022 at 7:13

1 Answer 1

Reset to default
3
$\begingroup$

There might be some difference between the training/validation and the testing data, causing performance gap. Another potential cause is that you tuned your model heavily on the validation set. This might make the model overfit to the validation set.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. But how should I fix the validation set? Size, distribution?--I didn't understand this: " heavily on the validation set. " $\endgroup$
    – Zehra N.
    Commented Mar 7, 2022 at 7:15
  • 1
    $\begingroup$ Sorry for my poor English. What I mean is that you might use the validation set to tune the hyperparameters too much. And I think you can first investigate the training/validation/testing sets. For example, plotting these sets in low dimensional space to check their distribution. If the testing set is too different from training/validation sets, then the low testing accuracy is not the model's fault. Or you can merge training and validation set together, and use k-fold cross validation on them. This would give you more robust results. $\endgroup$
    – stvhuang
    Commented Mar 7, 2022 at 9:32
  • $\begingroup$ Got it. Thank you $\endgroup$
    – Zehra N.
    Commented Mar 8, 2022 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.