# What is the role of feature engineering in statistical inference?

This may be a dumb question. I'm a recent college grad who is working in the area of predictive modeling and finding that there is a heavy emphasis on performing feature engineering. However, in most of my academic training in statistics, there was almost no mention of feature engineering and the like (besides arguments against discretizing/binning predictors) for the purpose of building inferential models. I was wondering why feature engineering plays a bigger role when doing predictive modeling as opposed to developing models for statistical inference. So...what is the role of feature engineering in statistical inference? (as opposed to the role of feature engineering in predictive modeling)

Based on the recent comment:

• By statistical inference, I mean any analysis where the main goal is to assess the relationship between a predictor and response variable.

• By predictive modeling, I mean any analysis where the main goal is to estimate Y or predict future values. (includes all ML techniques)

• You could perhaps edit the q. a bit to explain the distinction you're making between prediction & inference. Nov 16, 2016 at 20:22
• @aguy Thanks for the precision. I would have said however that prediction is always about events that have not occurred yet. From the work I have done in the field of ML, I do not think we can say that predictive modeling includes all machine learning techniques. Statistical inference is a broad term that embeds a large set of approaches and problems from classification and clustering to hypothesis testing to parameter estimation. Nov 16, 2016 at 22:20

I will try to illustrate the reason behind feature engineering in general, say I would like to analyze images.

When we design features, we have to keep in mind that they are a representation of the original data/image. Now, if I know which kind of information matter for the task I have to do, I need the features to reflect this.

For instance, if I would like to know the content of an image and I choose as feature the number of pixels in the image, it will not work, obviously. Now, if I choose to use the average intensity of the pixels across patches, I will be able to differentiate between a blue image and a white image. But maybe I want to know if an object is present in the image and this feature will be useless. So, I may consider the intensity gradients between the pixels and look at their variations (but then, I will not be able to say if my image is rather blue or white!).

There is no ideal feature, just features that are designed for a specific task and this task is only known from the person designing the entire framework: you! This is why feature engineering is important. However, research on the topic of features design is huge and for most tasks that you are working with, someone has already designed features that are proven to work great and you can just use them as is (or twist them a bit if needed).

Most efficient features are based on theories from various mathematical fields and their range of application is somehow narrow. Opposite, classifiers often have a broad range of applications and that's why I think the emphasis is often on them while in studying in academia.

• You may want to reread my question. I was asking about the role of feature engineering in statistical INFERENCE as opposed to PREDICTION, and why one places greater emphasis on it than the other
– AGUY
Nov 15, 2016 at 22:53
• @aguy First of all, classification models can be based on statistical inference. Second, I do not think that "feature engineering plays a bigger role when doing predictive modeling as opposed to developing models for statistical inference". Features are a representation of some data, you can see it as a projection of the data into a space which is more convenient for what you need to do, whatever it is (prediction, classification, clustering,..). Then the use of some models can have some constraints that need to be taken into account when designing features. Nov 16, 2016 at 22:11
• My last paragraph was actually corroborating your constatation: In class, we seldom hear about features design, but when starting to really work with any model (predictive, or other), we discover that they are crucial. Nov 16, 2016 at 22:13

As this Wiki article makes clear (https://en.wikipedia.org/wiki/Feature_engineering), feature engineering is a key step in machine learning, involving the generation and cultivation of a set of features or attributes that may prove empirically (not necessarily theoretically) useful in the prediction or classification of a target. Andrew Ng (and others) make much of expert, domain knowledge in the development of a set of features but given the multitude of transformations that can be applied to data to improve model fit, the massive numbers of features that are commonly analyzed and the "black-box" nature of many of the algorithms employed, domain knowledge hardly seems a priority.

For me, it's always useful to point out that inference vs prediction and classification can be viewed as separate domains, the former belonging to statistics and the latter the focus of machine learning. Obviously, there is much overlap in this terminology and these fields, i.e., they are by no means mutually exclusive. Broadly speaking, statistical inference involves expert, domain knowledge, careful specification of an hypothesis, a finite (small) set of attributes or features, coupled with an experimental design to test the hypothesis out -- classic scientific inquiry with the goal of driving insight and understanding relative to ground truth. ML prediction and classification, on the other hand, may or may not be hypothesis driven, may or may not have descriptive insight as a goal and may or may not have ground truth as a benchmark.

Predictors, dummy variables, or features are important in predictive modeling as they help capture genuine patterns in a data set and therefore make a better prediction since a model having a certain behavior will likely continue to have a certain behavior. And feature engineering helps capture this behavior.

Now for statistical inference based on your definition, you can already assess to an extent the relationship between predictor and response variable using exploratory analysis like scatterplots, correlation plots, correlograms, seasonal plots, lag plots. And further, strengthen your assessment by removing/adding the predictor from the features and evaluating the prediction.

So, feature engineering I would say is a crucial step in predictive modeling, and secondary in drawing statistical inference (since there are other methods to assess the relationship between available variables, looking into the historic data)

Feature engineering, broadly speaking, does at least 2 things.

First, you might clean, restructure, or transform your features in such a way that the useful information is enhanced and redundant or noise information is minimized. Perhaps you know that one category of people/products/widgets is totally irrelevant and remove them.

Second, you might create new features based on domain knowledge in your particular field. In this case, you actually add new information that was not there previously. In my own work, it's been these engineered features that provide the most utility.

This is probably difficult to teach, yet it's unfortunate your program overlooked this very important step.

• You may want to reread my question. I was asking about the role of feature engineering in statistical INFERENCE as opposed to PREDICTION, and why one places greater emphasis on it than the other
– AGUY
Nov 15, 2016 at 21:18