In the paper of fully convolutional networks semantic segmentation, the authors adopts up-sampling (de-convolutional network) to recover the feature maps with reduced dimensions (due to the multiple layers of down-sampling) to the original size.
If we do not do any down-sampling, i.e., using stride 1 in convolutional layer and pooling layer, and thus keep the image size after multiple layers of convolutions, can we just do the pixel-pixel semantic segmentation on the feature map of this layer without having to resort to upsampling as proposed by the paper.
I can imagine the performance can be worse, but I want to know whether this architecture really make sense? I think this question is also related to the importance of having "stride > 1", which is supposed to help getting abstract features(high level features).