7
$\begingroup$

I have seen it written in several papers and currently see it written in Deep Learning with Python by Francois Chollet that

Deep learning removes the need for feature engineering

What does this mean exactly? Does this apply to tabular data as well as perceptual data such as images?

$\endgroup$
1
  • $\begingroup$ it means in some applications deep learning can effectively construct the features itself using its neurons. this works well in many perceptive apps $\endgroup$
    – Aksakal
    Dec 30, 2019 at 14:49

2 Answers 2

4
$\begingroup$

Most deep learning models and their associated calibration processes are able to "perform" some simple feature engineering tasks like variable transformation and variable selection (it's difficult to speak about all models at the same time). Often it's more about how models are built than a specific action. This renders some basic feature engineering task unusefull.

For exemple a vanilla NN on tabular data will mostly be insensible to linear data transformation as each neuron rely on a linear predictor. The variable selection is somewhat done trough the weight calibration (may be with some regularisation) : if all the weights associated to a variable are low or 0, it is equivalent to its removal. Again, generally speaking, models refinements will enhance these properties.

However, for tabular data, I've found simple feature ingineering to be be crucial for the following reasons :

  • It's important to understand your data. I've found feature engineering to be an important step to have a look at your features, spot data quality problems and deal with them. Generally this will help you build better models. On some occasion understanding the data and building relevant features even lead me to go out of deep learning and build way simpler models.

  • Full deep learning necessitate a lot of computational ressources (computational power, memory). In general those ressources are limited and you will be better by removing poor predictors beforehand. Overall it may even translate to better performance as you will be able to build more sophisticated models with your restricted ressources. Non linear transformation of your data may also help the convergence of your calibration process by reducing the impact of 'outliers'.

  • Deep learning model are difficult to explain. Removing non-predictive features and building more predictive features trough feature engineering will often help you in that purpose. Whatever is your explainability solution, feature engineering will probably make it better. (Note that it is not necessarily true for more complex feature engineering steps)

$\endgroup$
3
  • $\begingroup$ Thanks. I agree with you that it would seem on tabular data, encoding domain knowledge into the feature space would be beneficial. However there are papers such as dl.acm.org/doi/pdf/10.1145/3292500.3330693?download=true which claim that on tabular data, an RNN model with no feature engineering beats an LGBM model with 7000 features. I find it surprising that even with extensive feature engineering, an RNN model beats it. $\endgroup$
    – cosmosa
    Jan 6, 2020 at 16:07
  • $\begingroup$ The article you link does not seems to be accessible. I don't know about what data they used, if they correctly set up their benchmark, and their performance metrics. With simplistic performance metrics you can find very very simple models : hdsr.mitpress.mit.edu/pub/f9kuryi8. that doesn't mean they are good overall, not enforceable. $\endgroup$
    – lcrmorin
    Jan 6, 2020 at 17:16
  • $\begingroup$ In case you're interested, here's the link with access: dl.acm.org/doi/10.1145/3292500.3330693 $\endgroup$
    – cosmosa
    Jan 6, 2020 at 18:17
4
$\begingroup$

To me, this is a (provocative, and intentionally so) restatement of the universal approximation theorem. Loosely speaking, the universal approximation theorem says that, given enough parameters, a neural network can get as close to a function as you want.

In this context, it means that you can let the neural network figure the composition of the feature extraction function and the regression function.

However, if you have insights into the features that would be important, you can give your neural network a break and do some of the work for it. Why make the network figure out that quadratic terms are important when domain knowledge (physics, chemistry, biology, etc) says that you know the quadratic term is important? You can, perhaps, use fewer parameters and put yourself at a lower risk of overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.