6
$\begingroup$

In some literature the convolution layers of convolutional neural networks have shared weights (e.g. see "shared weights" at deeplearning.net tutorial) but I already read some papers where the weights were non-shared and rather calculated like in "normal MLPs".

So, what is the advantage of shared weights for convolutional neural networks? Are there any pros and cons? Or is it just an architecture-based decision, like "we noticed, our network works better with shared weights"?

$\endgroup$
  • $\begingroup$ Can you please add some pointers to the papers "where the weights were non-shared and rather calculated like in "normal MLPs"? $\endgroup$ – Franck Dernoncourt Dec 29 '15 at 22:59
7
$\begingroup$

The main advantage of shared weights, is that you can substantially lower the degrees of freedom of your problem. Take the simplest case, think of a tied autoencoder, where the input weights are $W_{x} \in \mathbb{R}^d$ and the output weights are $W_{x}^T$. You have lowered the parameters of your model by half from $2d \rightarrow d$. You can see some visualizations here: link. Similar results would be obtained in a Conv Net.

This way you can get the following results:

  • less parameters to optimize,
  • which means faster convergence to some minima,
  • at the expense of making your model less flexible. It is interesting to note that, this "less flexibility" can work as a regularizer many times and avoiding overfitting as the weights are shared with some other neurons.

Therefore, it is a nice tweak to experiment with and I would suggest you to try both. I've seen cases where sharing information (sharing weights), has paved the way to better performance, and others, that made my model become significantly less flexible.

$\endgroup$
4
$\begingroup$

@iassael emphasized more on regularization effect as a result of reduced parameters, but I think better performance of weight sharing method is more about finding local features instead of global one. This reduces exponential possibilities to a linear scale or at least to a scale that can be more easily managed. Here is a simple example. Let's say we have an input with only 4 pixels and each pixel has only binary values '0' or '1'. there are 2^4 = 16 possible configuration for a global feature to learn. However, if a local feature, let's say it has only 1 pixel receptive field, is used, it is enough to learn 2 simple feature '0' and '1'. As receptive field size increases, number of feature needed to learnt also increases. As a result, local features reduces number of features that are needed to learnt. By convolving this local features on all input space it can be found where this features are present exaclty.

Let's apply the same analogy to an object detection test. If fully connected first layer attempts to extract all possible configurations of edges at given images, it needs to learn many combination of different edges which is plenty. However, if it tries to learn local features, number of possible edges greatly reduced to different edge orientations. Then via convolution it can reveals which location mostly activates a specific edge orientation.

$\endgroup$
  • $\begingroup$ in fact, by applying the same kernel independently of the location in the input image (which is guaranteed by making the kernel weights the same for all possible kernel centres), translational invariance is built into the neural network $\endgroup$ – Andre Holzner May 2 '16 at 19:32
2
$\begingroup$

A typical weight sharing technique found in CNN treats the input as a hierarchy of local regions. It imposes a general assumption (prior knowledge) that the input going to be processed by the network can be decomposed into a set of local regions with the same nature and thus each of them can be processed with the same set of transformations. With the prior assumption, we could reduce the amount of parameters in the network (as compared with a fully-connected network) and increase the network’s generalisation ability, given that this prior assumption is a correct one for the problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.