Why not set a static first layer in CNN?

From what I understand about convolutional neural networks.. Through a training process, convolutional filters start with random values that find random small/basic features. This result is pooled and then consumed by the next layer which finds combinations of these features. The process repeats, each time building a more complex feature representation of the original image. Then using back-propagation the filters are adjusted in a way that reduces error and steps closer towards the predictive goal.

My question is why don't we just set the first layer with static filters that find various angles of lines, and only train the rest? It seems as though the training process is just doing that anyways and we are making it work hard to find the same end result.

Note: I'm familiar with transfer learning, but I'm asking more about "artificially" set values.

My question is why don't we just set the first layer with static filters that find various angles of lines, and only train the rest?

Based on your description, what you are suggesting is called Extreme Learning Machine (ELM).

These are specific types of feed-forward neural networks that basically are different from the rest by having their hidden layers fixed (not trained), and instead just train to adjust the output layer. These layers are randomly initialized (or manually set based on heuristics) and don't change during training.

The idea is that, just like you say, by doing this one could train considerably faster than having to optimize for all hidden layers and nodes. According to some literature they tend to generalize better (as they are less prone to overfitting) and even outperform some other methods.

I would also guess that this principle could be extended or modified, so you leave an arbitrary number of layers fixed, while training the others; if we leave fixed the first layer and train for the others we get the specific situation you illustrated.

Furthermore, a way to implement this I can think of is to "freeze layers", a feature some APIs have already, like Tensorflow (check this question for several alternatives). The most straightforward option there mentioned is to set trainable=False on such variables.

It seems like a lot of work to me for minimal benefit. Having one more layer to backprop through when there may already be many tens of layers means that it doesn't help to hard-code the filters performance wise.

In addition, you only artificially limit yourself -- we know CNNs tend to learn gabor filters in the first layer -- but what if you have an unusual dataset? Or you're exploring a new architecture? Etc. At best, you can hope that by hard-coding in the first layer, you're not missing out on an even better solution in parameter-space.

And I think this is what you meant when you said transfer learning, but taking the pretrained weights from one model and using it to jumpstart the weights of another network is an effective strategy, and pretty similar to the static filters idea without the downsides.

The following quote from the 2002 review paper explains the role of preset initial layers in neural networks for image processing.

"According to Perlovsky, the key to restraining the highly Flexible learning algorithms for ANNs, lies in the very combination with prior (geometric) knowledge. However, most pattern recognition methods do not even use the prior information that neighbouring pixel=voxel values are highly correlated.

This problem can be circumvented by extracting features from images first, by using distance or error measures on pixel data which do take spatial coherency into account, or by designing an ANN with spatial coherency [LeCun1989] or contextual relations between objects in mind" [Egmont-Petersen2002].

• [LeCun1989] Y. LeCun, L.D. Jackel, B. Boser et al., Handwritten digit recognition—applications of neural network chips and automatic learning, IEEE Commun. Mag. 27 (11) (1989) 41–46.

• [Egmont-Petersen2002] M. Egmont-Petersen, D. de Ridder, H. Handels. Image processing with neural networks - a review, Pattern Recognition, Vol. 35, No. 10, pp. 2279-2301, 2002.