Why do neural networks need feature selection / engineering?

Question

Particularly in the context of kaggle competitions I have noticed that model's performance is all about feature selection / engineering. While I can fully understand why that is in the case when dealing with the more conventional / old-school ML algorithms, I don't see why this would be the case when using deep neural networks.

Citing the Deep Learning book:

Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.

Hence I always thought that if "information is in the data", a sufficiently deep, well-parameterised neural network would pick up the right features given sufficient training time.

Answer 1

• What if the "sufficiently deep" network is intractably huge, either making model training too expensive (AWS fees add up!) or because you need to deploy the network in a resource-constrained environment?

• How can you know, a priori that the network is well-parameterized? It can take a lot of experimentation to find a network that works well.

• What if the data you're working with is not "friendly" to standard analysis methods, such as a binary string comprising thousands or millions of bits, where each sequence has a different length?

• What if you're interested in user-level data, but you're forced to work with a database that only collects transaction-level data?

• Suppose your data are the form of integers such as $12, 32, 486, 7$, and your task is to predict the sum of the digits, so the target in this example is $3, 5, 18, 7$. It's dirt simple to parse each digit into an array and then sum the array ("feature engineering") but challenging otherwise.

We would like to live in a world where data analysis is "turnkey," but these kinds of solutions usually only exist in special instances. Lots of work went into developing deep CNNs for image classification - prior work had a step that transformed each image into a fixed-length vector.

Feature engineering lets the practitioner directly transform knowledge about the problem into a fixed-length vector amenable to feed-forward networks. Feature selection can solve the problem of including so many irrelevant features that any signal is lost, as well as dramatically reducing the number of parameters to the model.

Answer 2

The key words here are priors and scale. As a simple example, imagine you're trying to predict a person's age from a photograph. With a dataset of images and ages, you could train a deep-learning model to make the predictions. This is objectively really inefficient because 90% of the image is useless, and only the region with the person is actually useful. In particular, the person's face, their body and maybe their clothing.

On the other hand, you could instead use a pre-trained object detection network to first extract bounding boxes for the person, crop the image, and then pass it through the network. This process will significantly improve the accuracy of your model for a number of reasons:

1) All of the networks resources (i.e. weights) can focus on the actual task of age prediction, as opposed to having to first find the person first. This is especially important because the person's face contains useful features. Otherwise, the finer features that you need may get lost in the first few layers. In theory a big-enough network might solve this, but it would be woefully inefficient. The cropped image is also considerably more regular than the original image. Whereas the original image has a ton of noise, its arguable the discrepancies in the cropped image are much more highly correlated with the objective.

2) The cropped image can be normalized to have the same scale. This helps the second network deal with scaling issues, because in the original image, people can occur near or far away. Normalizing scale beforehand makes it so that the cropped image is guaranteed to have a person in it that fills the full cropped image (despite being pixilated if they were far away). To see how this can help scale, a cropped body that's half the width and height of the original image has 4x less pixels to process, and hence the same network applied to this image would have 4x the original network's receptive field at each layer.

For example, in the kaggle lung competition, a common theme in the top solutions was some kind of preprocessing on lung images that cropped them as much as possible and isolated the components of each lung. This is especially important in 3D images since the effect is cubic: by removing 20% of each dimension, you get rid of nearly half the pixels!

Answer 3

My intuition about this phenomenon is connected to the complexity of the model to be learned. A deep neural network can indeed approximate any function in theory, but the dimension of the parameter space can be really large, like in the millions. So, actually finding a good neural network is really difficult. I like to think about feature engineering as giving a head start to the algorithm, providing it some extra information regarding the data representation which is good enough in some sense. Of course, this is not a formal explanation, this question might be really hard to answer with scientific rigor.