I have often heard that Deep Learning Models (i.e. Deep Neural Networks) are automatically performing feature engineering within the network themselves. This is contrasted with traditional statistical and Machine Learning models where the feature engineering is typically done prior to training the model:
Apparently, the operations that Neural Networks perform during the training phase are equivalent to "searching for meaningful combinations of variables that produce better performance results on the training data". I have heard that this is in some way similar to Principal Component Analysis (PCA) and Kernel Methods, seeing as these methods combine many different existing features into new features that have "more meaningful representation".
In this previous post (Why do neural networks need feature selection / engineering?), I read about this ability of Deep Neural Networks to "automatically perform feature engineering" :
Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.
However, in the end - I still do not understand how Neural Networks during the training phase (i.e. gradient descent, weight update, backpropagation) are "automatically performing (some degree of) feature engineering". How exactly are Neural Networks doing this?