What happens if weight sharing is avoided in convolutional neural networks?

Question

I understand that one of the advantages of convolutional layers over dense layers is weight sharing. Assuming that memory consumption is not a constraint, would a CNN work better if a different kernel is trained for each patch of input? In other words, if instead of training a single kernel for all input patches, I train a different kernel for each patch, should I expect to get better or worse results?

Answer 1

The consequences are that you need to learn each feature separately for each part of the image. So if you used it to predict birds and your data contained only blue birds in upper left corner and only the white birds in lower right, then you wouldn't be able to classify in the lower right corner blue bird as one despite that you had them in your dataset.

Answer 2

What you described is called "Locally connected layers" and it is a trade-off between convolutional layers and fully connected ones, as the following figure [1] visualizes:

It has much less parameters than a fully-connected layer but much more than a convolution layer. Whether you will get better or worse results depends entirely on the problem you are trying to solve. Look for papers using locally connected layers to see for which problems is this approach suitable.

---

[1]: Chen, Y. et al., 2015: Locally-Connected and Convolutional Neural Networks for Small Footprint Speaker Recognition.