I have often heard that Deep Learning Models (i.e. Deep Neural Networks) are automatically performing feature engineering within the network themselves. This is contrasted with traditional statistical and Machine Learning models where the feature engineering is typically done prior to training the model:

enter image description here

Apparently, the operations that Neural Networks perform during the training phase are equivalent to "searching for meaningful combinations of variables that produce better performance results on the training data". I have heard that this is in some way similar to Principal Component Analysis (PCA) and Kernel Methods, seeing as these methods combine many different existing features into new features that have "more meaningful representation".

In this previous post (Why do neural networks need feature selection / engineering?), I read about this ability of Deep Neural Networks to "automatically perform feature engineering" :

Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.

However, in the end - I still do not understand how Neural Networks during the training phase (i.e. gradient descent, weight update, backpropagation) are "automatically performing (some degree of) feature engineering". How exactly are Neural Networks doing this?



1 Answer 1


"Neural Networks Automatically Do Feature Engineering"

it's a very general statement, and it's somewhat not invalid in some domains. suppose you're modeling $y=ae^{bt}$ process. the best way to do it would be to log-transform then differentiate the data and model $\frac{\Delta\ln y}{\Delta t}=b$, basically a constant. Done.

The trouble is that you may have a lot of potential factors that enter into a very complex set of relationships $y=f(x_1,\dots,x_n)$, so it's not feasible to spend your lifetime trying to get them in some better form $g(y)=h(m_1(x_1),\dots,m_n(x_n))$. you hope that this $h(.)$ function is simple in structure, and easy to approximate from data.

So, instead you throw all variables into NN hoping that with its trillions of neurons it will be able to get the $f(.)$ function's structure without explicit feature engineering. this works in some cases.

conceptually, this looks like a function table. those of us who're too old, remember the books published with function tables. NN is essentially such a table, where for every $t$ input it memorizes what should be $y$. so for the exponential function in my example you need to do a lot of memorization if the domain is wide.

Now, why would you still need feature engineering? To save neurons in NN. so, in the example I gave the NN can be very small just to estimate parameters $a,b$, and exponentiation and log be done outside of it if you do feature engineering. instead if you try to fit the original model $y=f(x)$ without using knowledge that there's exponent inside, then NN will be much larger.

  • $\begingroup$ Thank you so much for your answer! I have never heard about a "function table" - is this similar to a trigonometry table or a logarithm table? $\endgroup$
    – stats_noob
    Jan 31, 2022 at 4:36
  • $\begingroup$ "So, instead you throw all variables into NN hoping that with its trillions of neurons it will be able to get the f(.) function's structure without explicit feature engineering. this works in some cases." - are we only "hoping" ? do we understand why this works in some cases and does not work in some cases? $\endgroup$
    – stats_noob
    Jan 31, 2022 at 4:37
  • $\begingroup$ Yes, I mean a table with values of y for given x. In fact inside the cpu for log and other transcendent functions there are tables used to interpolate functions. People spent a lot of time coming up with them. There used to be entire books on function studies like iucat.iu.edu/iuk/787296 $\endgroup$
    – Aksakal
    Jan 31, 2022 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.