I submitted a paper that uses an auto-encoder network with several convolutional layers in both the encoder and the decoder and a fully connected layer (FCL) in between. Besides the FCL being useful or not, a reviewer commented that having a FCL far from the output has severe effects on the gradient and could greatly slow down the learning. I think this might be true but I could not find any publications that back it up. Is it true? Are there any studies conducted on this topic?
I've never heard of such a thing. In fact many fully convolutional autoencoder models do have a bottleneck layer which has spatial dimensions 1x1, which is equivalent to saying that the layer just before and just after are fully connected.
I could imagine that such a layer could slow down learning if it has a massive number of parameters, since FC layers tend to have more parameters than convolutional ones. It could also hurt the translational equivariance of the network. But to the best of my knowledge there's nothing which affects the quality of the gradients.