I have seen it written in several papers and currently see it written in Deep Learning with Python by Francois Chollet that

Deep learning removes the need for feature engineering

What does this mean exactly? Does this apply to tabular data as well as perceptual data such as images?

  • $\begingroup$ it means in some applications deep learning can effectively construct the features itself using its neurons. this works well in many perceptive apps $\endgroup$ – Aksakal Dec 30 '19 at 14:49

Most deep learning models and their associated calibration processes are able to "perform" some simple feature engineering tasks like variable transformation and variable selection (it's difficult to speak about all models at the same time). Often it's more about how models are built than a specific action. This renders some basic feature engineering task unusefull.

For exemple a vanilla NN on tabular data will mostly be insensible to linear data transformation as each neuron rely on a linear predictor. The variable selection is somewhat done trough the weight calibration (may be with some regularisation) : if all the weights associated to a variable are low or 0, it is equivalent to its removal. Again, generally speaking, models refinements will enhance these properties.

However, for tabular data, I've found simple feature ingineering to be be crucial for the following reasons :

  • It's important to understand your data. I've found feature engineering to be an important step to have a look at your features, spot data quality problems and deal with them. Generally this will help you build better models. On some occasion understanding the data and building relevant features even lead me to go out of deep learning and build way simpler models.

  • Full deep learning necessitate a lot of computational ressources (computational power, memory). In general those ressources are limited and you will be better by removing poor predictors beforehand. Overall it may even translate to better performance as you will be able to build more sophisticated models with your restricted ressources. Non linear transformation of your data may also help the convergence of your calibration process by reducing the impact of 'outliers'.

  • Deep learning model are difficult to explain. Removing non-predictive features and building more predictive features trough feature engineering will often help you in that purpose. Whatever is your explainability solution, feature engineering will probably make it better. (Note that it is not necessarily true for more complex feature engineering steps)

  • $\begingroup$ Thanks. I agree with you that it would seem on tabular data, encoding domain knowledge into the feature space would be beneficial. However there are papers such as dl.acm.org/doi/pdf/10.1145/3292500.3330693?download=true which claim that on tabular data, an RNN model with no feature engineering beats an LGBM model with 7000 features. I find it surprising that even with extensive feature engineering, an RNN model beats it. $\endgroup$ – cosmosa Jan 6 '20 at 16:07
  • $\begingroup$ The article you link does not seems to be accessible. I don't know about what data they used, if they correctly set up their benchmark, and their performance metrics. With simplistic performance metrics you can find very very simple models : hdsr.mitpress.mit.edu/pub/f9kuryi8. that doesn't mean they are good overall, not enforceable. $\endgroup$ – lcrmorin Jan 6 '20 at 17:16
  • $\begingroup$ In case you're interested, here's the link with access: dl.acm.org/doi/10.1145/3292500.3330693 $\endgroup$ – cosmosa Jan 6 '20 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.