I am training U-Net models for two-class semantic segmentation (foreground/background). I have tested different depths of the U-Net along with different number of filters in the first conv-layer (the ones further down follow from doubling them at each binning step). I maintained the size of the filters fixed at 3x3 and the number of conv-layers per conv-block fixed at 3 (see the image below for what I mean by U-Net "conv-block" and levels; this is just for illustration: the indicated numbers/output-layer/etc do not correspond 1:1 to my models).
I have now found that models with two U-Net conv-blocks/levels perform the same to models with e.g. 3 conv-blocks/levels, while maintaining approximately equal number of trainable parameters (which I achieve by changing the number of filters in the first layer accordingly). Overfitting is similar between both as well.
Occam's razor suggests, that I should use the least complex model that accurately describes my data. For me it is not obvious, which one I should pick in this case, since both models have the same number of parameters.
I know that deeper models have higher "representational power/learning capacity", which is why we train deep models after all. But is this identical to being more complex?
My hunch tells me "yes", but I would like a more theoretically grounded understanding/argumentation of "why". Obviously, the scope of this question goes beyond my example of CNNs.
Any links to relevant (scientific) literature would be much appreciated!
REF: Image adapted from here: