"Neural Networks Automatically Do Feature Engineering" - How?

Question

I have often heard that Deep Learning Models (i.e. Deep Neural Networks) are automatically performing feature engineering within the network themselves. This is contrasted with traditional statistical and Machine Learning models where the feature engineering is typically done prior to training the model:

Apparently, the operations that Neural Networks perform during the training phase are equivalent to "searching for meaningful combinations of variables that produce better performance results on the training data". I have heard that this is in some way similar to Principal Component Analysis (PCA) and Kernel Methods, seeing as these methods combine many different existing features into new features that have "more meaningful representation".

In this previous post (Why do neural networks need feature selection / engineering?), I read about this ability of Deep Neural Networks to "automatically perform feature engineering" :

Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.

However, in the end - I still do not understand how Neural Networks during the training phase (i.e. gradient descent, weight update, backpropagation) are "automatically performing (some degree of) feature engineering". How exactly are Neural Networks doing this?

Thanks!

Answer 1

"Neural Networks Automatically Do Feature Engineering"

it's a very general statement, and it's somewhat not invalid in some domains. suppose you're modeling $y=ae^{bt}$ process. the best way to do it would be to log-transform then differentiate the data and model $\frac{\Delta\ln y}{\Delta t}=b$, basically a constant. Done.

The trouble is that you may have a lot of potential factors that enter into a very complex set of relationships $y=f(x_1,\dots,x_n)$, so it's not feasible to spend your lifetime trying to get them in some better form $g(y)=h(m_1(x_1),\dots,m_n(x_n))$. you hope that this $h(.)$ function is simple in structure, and easy to approximate from data.

So, instead you throw all variables into NN hoping that with its trillions of neurons it will be able to get the $f(.)$ function's structure without explicit feature engineering. this works in some cases.

conceptually, this looks like a function table. those of us who're too old, remember the books published with function tables. NN is essentially such a table, where for every $t$ input it memorizes what should be $y$. so for the exponential function in my example you need to do a lot of memorization if the domain is wide.

Now, why would you still need feature engineering? To save neurons in NN. so, in the example I gave the NN can be very small just to estimate parameters $a,b$, and exponentiation and log be done outside of it if you do feature engineering. instead if you try to fit the original model $y=f(x)$ without using knowledge that there's exponent inside, then NN will be much larger.