I have multiclass classification dataset and I am using Deep nets for the classification task. To explain the problem, let's assume that I have 5 classes to classify. No matter what I try, be it hyperparameter tuning or trying out different architectures including CNN and RNN, I am not able to separate 2 out of these 5 classes. I mean it is classifying, but classification accuracy between these 2 classes is not going above 60 %.
What could be the reasons for these? Are these two classes truly inseparable? It only takes about 10 epochs for the training accuracy to hit 100 % but the validation accuracy never improves after 55 to 60 %. The models overfit so easily, still no matter what I try to prevent overfitting like early stopping, Dropout etc nothing works, it either overfits or under fits.
At this point, I assume that the train and test datasets are not similar, I mean they both could be from different distributions. How can I know if my train and test data follows the same distribution? More generally, how to find out that the train and test datasets are somewhat similar? If the classes are truly difficult to separate, then the training task should also be difficult, isn't it?
