33
$\begingroup$

I often see people create new features based on existing features on a machine learning problem. For example, here : https://triangleinequality.wordpress.com/2013/09/08/basic-feature-engineering-with-the-titanic-data/ people have considered the size of a person’s family as a new feature, based on the number of brothers, sisters, and parents, which were existing features.

But what the point of this ? I don't understand why the creation of correlated new features is useful. Isn't it the job of the algorithm to do that on its own ?

$\endgroup$
1

3 Answers 3

39
$\begingroup$

The most simple example used to illustrate this is the XOR problem (see image below). Imagine that you are given data containing of $x$ and $y$ coordinated and the binary class to predict. You could expect your machine learning algorithm to find out the correct decision boundary by itself, but if you generated additional feature $z=xy$, then the problem becomes trivial as $z>0$ gives you nearly perfect decision criterion for classification and you used just simple arithmetic!

XOR problem

So while in many cases you could expect from the algorithm to find the solution, alternatively, by feature engineering you could simplify the problem. Simple problems are easier and faster to solve, and need less complicated algorithms. Simple algorithms are often more robust, the results are often more interpretable, they are more scalable (less computational resources, time to train, etc.), and portable. You can find more examples and explanations in the wonderful talk by Vincent D. Warmerdam, given on from PyData conference in London.

Moreover, don't believe everything the machine learning marketers tell you. In most cases, the algorithms won't "learn by themselves". You usually have limited time, resources, computational power, and the data has usually a limited size and is noisy, neither of these helps.

Taking this to the extreme, you could provide your data as photos of handwritten notes of the experiment result and pass them to the complicated neural network. It would first learn to recognize the data on pictures, then learn to understand it, and make predictions. To do so, you would need a powerful computer and lots of time for training and tuning the model and need huge amounts of data because of using a complicated neural network. Providing the data in a computer-readable format (as tables of numbers), simplifies the problem tremendously, since you don't need all the character recognition. You can think of feature engineering as a next step, where you transform the data in such a way to create meaningful features so that your algorithm has even less to figure out on its own. To give an analogy, it is like you wanted to read a book in a foreign language, so that you needed to learn the language first, versus reading it translated in the language that you understand.

In the Titanic data example, your algorithm would need to figure out that summing family members makes sense, to get the "family size" feature (yes, I'm personalizing it in here). This is an obvious feature for a human, but it is not obvious if you see the data as just some columns of the numbers. If you don't know what columns are meaningful when considered together with other columns, the algorithm could figure it out by trying each possible combination of such columns. Sure, we have clever ways of doing this, but still, it is much easier if the information is given to the algorithm right away.

$\endgroup$
4
  • $\begingroup$ Thank you, it's very well explained and very interesting ! $\endgroup$ Jun 7, 2018 at 11:15
  • $\begingroup$ I find your answer very good but I think that it is better to qualify it a bit. Specifically, I think that feature engineering in terms of interaction terms does not simplify things that much for algorithms such as RFs (although it can still help) since RFs capture the interaction terms to a certain extent by themselves. Let me know if I am missing something important. But yes in general feature engineering simplifies and helps things quite a lot. $\endgroup$
    – Outcast
    Jun 13, 2019 at 15:54
  • $\begingroup$ @PoeteMaudit it is true that decision tree (and so RF) models one big interaction, but it's a certain kind of interactions, not every possible interaction. Moreover, feature engineering is not only about interactions. $\endgroup$
    – Tim
    Jun 13, 2019 at 16:20
  • $\begingroup$ Surely I agree with your points. Firstly, that RFs models a certain kind of interactions (you mean mainly x*y?). Secondly, that feature engineering is certainly not only about interactions. $\endgroup$
    – Outcast
    Jun 13, 2019 at 17:14
15
$\begingroup$

Well, if you plan to use a simple, linear classifier, it makes perfect sense to generate new features which are a non-linear function of the existing ones, specially if your domain knowledge indicates you the resulting feature will be meaningful and informative. Note that a linear classifier cannot consider those complex features unless you explicitly provide them.

Ideally, If you use a sufficiently powerful nonlinear classification algorithm it should be able to create a decision boundary which considers arbitrary non-linear transformations of the input features if they are informative for classification. However, in practice most non-linear classifiers just look at some type of transformations. For instance, a polynomial kernel SVM will consider polynomial interactions between features, but maybe a more informative feature can be created by applying other types of transformations...

In short, if domain knowledge indicates that a hand-crafted non-linear combination of features might be informative, it makes sense to add that into the existing set of features.

$\endgroup$
3
  • $\begingroup$ Ok I understand ! But how can I know if my features I created are meaningful and informative ? I mean, problems can have counter-intuitive explainations. From what I understand, creating features is a way to steer the algorithm in one direction, to save time on its predictions. So in a few ways, we are influencing the algorithm. How to verify that we influence correctly? $\endgroup$ Jun 7, 2018 at 11:20
  • 2
    $\begingroup$ Well, that is were the domain knowledge comes in. For instance, if you are designing a medical diagnosis system, you may talk to doctors and ask them what information they use to diagnose a patient. For instance, if one doctor says "If the patient has both fever and headaches, then he is likely to have a flu". That sentence might indicate that including a new flag feature which indicates the presence of both fever and headaches might be a good idea (specially if you use a linear classifier, which cannot "see" polynomial interaction features by itself). $\endgroup$ Jun 7, 2018 at 11:27
  • 1
    $\begingroup$ Nevertheless, this is more of an art than a science. At the end, you need to validate the effectiveness of your hand-crafted features by testing your models on a test set with an unbiased experimental protocol... $\endgroup$ Jun 7, 2018 at 11:28
4
$\begingroup$

It is true that some of the machine learning models have the ability to handle the non-linearity and interaction between variables, however, depends on the situation, I see three reasons it becomes necessary.

  1. Some models like linear regression don't handle non-linearity automatically, in that case, you need to create extra features to help. For example below: if you have the following dataset that all the $Y = 1$ of the target variable clustered at the center of a circle like area.

Circle Boundary

If you are given only two features, $x_1$ and $x_2$. A simple linear model of $y = x_0 + c_1x_1 + c_2x_2$ will not find any way to classify the target variable. So, instead, you need new quartic features to capture the non-linearity: $y = x_0 + c_1x_1^2 + c_2x_2^2$.

  1. If you know in advance that some features (from business knowledge or experience), it may help create them to speed up the runtime of the model and make it easy for your model. For example, in your example of the Titanic data and if you are using a decision tree classification model. If you know that old ladies (age & gender) are more likely to survive, by creating a single feature that captures the information, your tree can make one split on the new variable instead of making two split on the two variables. It may speed up the computation time if you know in advance that the feature is significant.

  2. In the real world, you won't get a single dataset like Kaggle provides. Instead, you get information from all over the place. For example, if you want to predict customer attrition for an online retail company like Amazon, you have customer demography info, purchase transaction info. You need to generate a lot of feature from different sources, in this case, You will find a lot of useful features can be obtained/aggregated from the transaction level. As Andrew Ng puts it: Often times, the ability to do feature-engineering defines the success or failure of a machine learning project.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.