Feature extraction in deep neural networks

Question

From many definitions that I read, I concluded that a DNN (deep neural network) is an ANN (artificial neural network) that have more than one hidden layer.

Knowing that CNN (convolutional neural network, a kind of a DNN) includes a stage of feature extraction (through convolution operations then pooling), my question is:

Does any DNN naturally considers feature extraction? For example, if we assume a simple architecture that includes three fully-connected hidden layers. I think not. If it is really not, how a step of feature extraction can be introduced within such a DNN? Should I necessarily introduce convolution like in CNN?

Answer 1

If your problem is "nice enough" that CNNs can do a good job on it, congratulations! Most of the work has been abstracted away! But if you're unlucky enough to be working on any of the myriad of problems that don't naturally lend themselves to CNNs or similar "feature-free" neural networks, then you'll have to spend a lot of time figuring out how to collect and represent your data in a useful way.

One example is computer security. A portable executable file is billions or more bytes and usually contains heterogenous data types. Feature engineering, the painstaking process of measuring various attributes of the file, is critically important to representing this data in a format that is useful and yields strong results. Feature-free attempts at analyzing PE files have not yet achieved parity with handcrafted feature vectors. (This might happen in the future, but it hasn't happened yet.)

The discussion at Why do neural networks need feature selection / engineering? is also helpful here.

Answer 2

Before deep NNs, people used to engineer features manually. For example, to learn an image classifier, you could preprocess your images using Gabor filters or use SIFT, etc. These algorithms do not learn features, they are invented by people as general purpose tools to extract features from raw data. By features, we mean data that is lower-dimensional than the raw data itself (that is why CNNs use pooling, to downsample the data) and that encode higher-level characteristics of the datapoint, for example the color or the edges of an object in an image.

Deep learning is an approach to machine learning that does away with these fixed preprocessing step and learn features. The idea is that by using feature extractors that are learned specifically for a task, the features suit the task better and the overall performance can be improved.

Convolutions in themselves are just a building block. If you remove convolutions and use fully-connected layers in a DNN, you still have a feature extraction step.

You could consider that if you have a $n$-layers DNN classifier, the $n-1$ layers constitute a feature extractor. The last layer is a linear classifier that operate on these complex, task-specific, learned features.