# Why does locally connected layer work in convolutional neural network?

Locally connected pixels are highly correlated in images. In machine learning, using SVM or other classifiers we would want our features to be as unrelated as possible.

However, in CNNs why do we want exactly the opposite thing? We are focusing on highly correlated features (i.e. pixels) to convolve with our weight vector (say a 5x5 filter)?

I understand its usage to reduce the no. of parameters, but I do not clearly understand that why local connections should work? Basically what is the motivation behind considering local connections in images?

In a convolution layer the filter has an output depth parameter. So a 5x5 filter is actually 5x5xd, d being the output depth. This essentially means that there are 'd' different filters, each of which will (learn to) have different weights. Each filter will detect the presence of a particular pattern across a feature map (e.g. an input image) but different filters will likely learn to detect unrelated features. For example one filter might have learnt to find vertical lines while a second one might have learnt to detect slanted lines at 45 degrees.

So the application of a convolution layer will in fact generate new high-level features (like straight lines) which are unrelated, given low level features (pixels in a neighborhood) that are likely to be highly correlated.

This is because if a filter is successful in extracting a useful feature from a small portion of the image, we would like to extract the same feature from other parts of the image. This is related to this fact that we would like to extract translation-invariant features from the image, that is we want to extract features from the image which do not change when the objects in the image are moved to another place in the image. This is not specific to the CNNs but is the common paradigm in machine vision.

You could see the Convolutional layers as a dimension reduction technique. Indeed, nearby pixels share a lot of covariance and ideally the features for a machine learning approach are independent.

If the convolutional operator is replaced by a specific convolutional operator were all the weights are $1/d^2$ (i.e. the average) it effectively reduces the dimensions of the picture. This operation however is rather simple and not tuned to our specific learning goal. By learning the weights of a conv operator we can 'tune' our 'average' to our learning goal.

The conv operation works that well for 2d and 3d inputs because it gives structure to this dimension reduction. It focuses on learning the variance of nearby pixels. By removing the variances of nearby pixels the latter fully-connected layers can be more effective.

As per Ian Goodfellow et al. from deeplearningbook:

Locally connected layers are useful when we know that each feature should be a function of a small part of space, but there is no reason to think that the same feature should occur across all of space. For example, if we want to tell if an image is a picture of a face, we only need to look for the mouth in the bottom half of the image. It can also be useful to make versions of convolution or locally connected layers in which the connectivity is further restricted, for example to constrain each output channel i to be a function of only a subset of the input channels l