Should I be stepping down high a dimensional embedding when predicting low dimensional output

Question

I'm using a ResNet-50 pretrained on ImageNet as a starting point for various image classifiers. Because the pretrained model has 1001 outputs, I have added a single dense layer with output size 500 in between the ResNet dense layer and my output layer, which differs depending on the type of classifier I'm training.

In many cases, I will train a binary classifier, so the network will look like the following.

ResNet-50 (1001) -> Dense(500) -> Dense(2)

This seems to produce reasonable results, but I can't help but think that these models would benefit from stepping down the output dimensions so that the difference in size between the final hidden layer and the output layer isn't so stark.

So instead, perhaps something like the following, but with some dropout as well. Note that I am not disclosing other details related to what residual blocks are being trained, etc. This is just a general process that I'm trying to wrap my head around.

ResNet-50 (1001) -> Dense(500) -> Dense(250) -> Dense(75) -> Dense(10) -> Dense(2)

Answer 1

I don't think there's necessarily a one size fits all answer for this.

What you describe in a way fits what many neural networks used to do (see e.g. VGG-19/-16). However, note that the second approach has simply more trainable parameters through the extra layers. As a result, I would expect it to do better, if you have enough data. I assume you have limited data (and are thus doing transfer learning), so it's hard to know whether in your specific case this helps (extra capacity needed to learn all the generalizable things about your training data) or harms (too many parameters versus samples leading to overfitting) performance on your test set.

The best way to find out is likely to do a train-validation-test split, optimize the architecture vs. the validation data, retrain on train+validation with the chosen architecture and to then evaluate on test.