I trained a resnet101 model using 900 samples / class (50 classes in total ), and chose the best model based on best accuracy using a separate validation set of 100 samples / class, throughout the whole training train/val accuracies were very close, which mean i am not over-fitting the train dataset(?), but the accuracy of a hold on test set of 100 samples / class was lower by 6~10% compared to validation accuracy, what could be the reason? NB: train/ val/test sets were sampled from the same distribution
You may be "overfitting to the validation set" at this step:
chose the best model based on best accuracy using a separate validation set
Once you tweak your model, or choose among multiple competing models based on the validation set, it is not a true validation set any more - it effectively becomes part of the training set, albeit in a somewhat obfuscated way.
Look at it in this way: suppose you train an astronomical number of models, including all your potential predictors, but also transformations, and completely unrelated predictors like sunspot numbers or frequencies of Mongolian throat singers. Then you apply all your models to the validation set and choose the one model that performs best on the validation set. This one happens to include an interaction term between last year's precipitation in Nowhere, OK and the hemlines at this year's Paris fashion shows - simply because you are fitting noise.
Would you expect this model to perform well on holdout test data, simply because it "worked" best on the validation set?
Unfortunately, there really is no good way around this dilemma. This can even happen with cross-validation and regularization or tree pruning. Differences between accuracy on different test and validation sets are one potential hint that something like this is going on, though. Your best approach would be to be parsimonious in your initial selection of candidate models, and collecting a lot of test data so you can use a "layered" approach with multiple iterated validation and test sets.