2
$\begingroup$

It is my intuition that unrestricted / unshared weights encourage the network to capture more invariances than in the case of convolutions, where the weights are shared. (These capture mostly translational invariances). See this and this.

If sharing weights actually results in less invariant feature detectors, why are convolutional layers preferred? Is it just because they're faster?

$\endgroup$
1
$\begingroup$

Convolutions are preferred because they capture global translational invariances.

A layer with Local receptive fields (LRF) which does not share weights across the entire input structure does not capture them.

In the second link that you cite, the proposed model does not need such invariance because it is applied to a face recognition problem for which the images are centered.

In particular, each LRF is associated to the same kind of region for each image. A LRF in the hair would capture information very different from a LRF in the eye for example, thus it makes little sense to make them share weights.

Certainly this implementation is clever for this type of dataset but is at a greater computational cost.

In the first link, there is actually weight sharing. The parameter $k$ controls the number of steps away the weights of a LRF are shared with another LRF, allowing them to capture the wanted translational invariances as well as other additional invariances.

They showed that their model works best for $k=2$. In this case, the number of parameters is only doubled (a LRF has its weights shared with half of all LRF).

A good explanation about parameter sharing can be found on wikipedia's cnn article.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.