Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have seen it written in several papers and currently see it written in Deep Learning with Python by Francois Chollet that
Deep learning removes the need for feature engineering
What does this mean exactly? Does this apply to tabular data as well as perceptual data such as images?
To me, this is a (provocative, and intentionally so) restatement of the universal approximation theorems. Loosely speaking, the universal approximation theorems say that, given enough parameters, a neural network can get as close to a decent function as you want.
In this context, it means that you can let the neural network figure the composition of the feature extraction function and the regression function.
However, if you have insights into the features that would be important, you can give your neural network a break and do some of the work for it. Why make the network figure out that quadratic terms are important when domain knowledge (physics, chemistry, biology, etc) says that you know the quadratic term is important? You can, perhaps, use fewer parameters and put yourself at a lower risk of overfitting.
Most deep learning models and their associated calibration processes are able to "perform" some simple feature engineering tasks like variable transformation and variable selection (it's difficult to speak about all models at the same time). Often it's more about how models are built than a specific action. This renders some basic feature engineering task unusefull.
For exemple a vanilla NN on tabular data will mostly be insensible to linear data transformation as each neuron rely on a linear predictor. The variable selection is somewhat done trough the weight calibration (may be with some regularisation) : if all the weights associated to a variable are low or 0, it is equivalent to its removal. Again, generally speaking, models refinements will enhance these properties.
However, for tabular data, I've found simple feature ingineering to be be crucial for the following reasons :
It's important to understand your data. I've found feature engineering to be an important step to have a look at your features, spot data quality problems and deal with them. Generally this will help you build better models. On some occasion understanding the data and building relevant features even lead me to go out of deep learning and build way simpler models.
Full deep learning necessitate a lot of computational ressources (computational power, memory). In general those ressources are limited and you will be better by removing poor predictors beforehand. Overall it may even translate to better performance as you will be able to build more sophisticated models with your restricted ressources. Non linear transformation of your data may also help the convergence of your calibration process by reducing the impact of 'outliers'.
Another point is when the instance are not independant, both in terms of features across instances or in terms of targets. Feature enginnering (mean by group for exemple) allows to build features across many instance. Target encoding / engineering / imputation may help better capture the interdependance of targets. Those are features that can drastically improve the model over 'base' features, as they bring new information to them.
Deep learning model are difficult to explain. Removing non-predictive features and building more predictive features trough feature engineering will often help you in that purpose. Whatever is your explainability solution, feature engineering will probably make it better. (Note that it is not necessarily true for more complex feature engineering steps)
