Feature extraction definition

Question

I have difficulty understanding the concept of feature extraction since there are two main ways to describe it.

1. The first one refers to mapping the raw data into a vector in R^d or the translation of raw data into the inputs required from a machine learning algorithm. For instance, from text, we can tokenize to get vectors, or extract information from raw pixels from images, etc.

2. The second definition extracts new features from already existing features and refers to the process of transforming o projecting a space composing of many dimensions into a space of fewer dimensions, such as with PCA. Where extracted features are a combination of the original features.

Although both definitions are similar, the first one is more related to a "construction of features" while the other implies transform already existing features. How could I call for instance to the process of using human expertise to know what to measure from objects and construct a set of features?

Answer 1

The difference between the two definitions (construction of features from raw data vs. the extraction of new information from already created features) is minimal, but there is an obvious implication that the former takes place earlier in the process of cleaning and preprocessing data and is thus giving a lot more information and structure to data that might otherwise be ignored. But the goals implied within both definitions are the same: create a variable with information that an eventual algorithm could use to distinguish instances of different labels from one another.

For your specific question, I would count it as feature extraction because you are adding more information and structure to data with human expertise than you would otherwise. But, hypothetically, you would also be doing some feature extraction by transforming the variables into some more meaningful format. What ultimately matters is how your model makes use of the information (and it is definitely part of building your model).

Answer 2

Both of these definitions I'd say are part of the process of feature engineering/feature extraction. The first is just more preliminary, as your model gets more complex and you tweak it, you may utilise the tools of the second such as transformations or PCA (technically a linear transformation). A transformation in a simple case could involve something like squaring one of your variables thereby making your model polynomial for example, while PCA will help remove redundancy in your data, reducing the number of features.

What you're describing as a process of choosing what to measure before presumably you even generate your data set sounds a bit like experimental design. If you were doing A/B testing for example followed by building a predictive model then the experimental design would come first and with it choice of variables to measure. Further feature engineering would follow once you had the data. So at the very least the process you're describing would happen prior to the other two definitions of feature extraction. (Note I'm using feature engineering/feature extraction interchangeably here whereas sometimes engineering is the broader category while extraction a subset)