I need a high-level taxonomy of dimensionality reduction approaches. I need your help to unify what I have found:
1) according to wikipedia, dimensionality reduction can be divided into: feature selection (select subset of original variables) and feature extraction (transform the original space into lower-dimensional space).
2) according to this post on Cross Validated, Two broad categories of dimensionality reduction , there is parametric and non-parametric dimensionality reduction. For example, PCA is parametric, with linear mapping.
3) according to https://www.analyticsvidhya.com/blog/2015/07/dimension-reduction-methods/ , the most common methods are:
- Missing values - this is basically feature selection, we decide if we should remove some of the existing variables, due to too many missing values. This is basically Feature Selection.
- Low variance - remove a variable with very low variance or maybe even constant values for all training examples, because it can't tell us much about the variance in other variables. This is basically Feature Selection.
- Decision Trees - can be used to tell us which variables are significant, so we know which variables to keep and which to remove. Again, this is Feature Selection.
- Random Forests - quote: "to select a smaller subset of input features." Again, this is Feature Selection.
- High Correlation - remove a variable if it is highly correlated with another variable, because more or less, they carry the same information. Again, this is Feature Selection.
- Backward Feature Elimination - quote: "Then, identifying variables whose removal has produced the smallest increase in the SSR and removing it finally, leaving us with n-1 input features.". Again, this is Feature Selection.
- Factor Analysis - group highly correlated variables into a so called factor variable. This is not feature selection. Here we combine more variables into one, and in that way we reduce the number of dimensions. Not sure what to call this, it is neither feature selection, nor feature extraction.
- PCA - this is Parametric Feature Extraction. The original variables are transformed by linear function f into new variables.
CONCLUSION: We can divide dimensionality reduction into:
Feature Selection -remove one or more of the original variables according to some criteria.
Combine more variables into one. Not sure what to call this one ???
Feature Extraction - a mapping transforms the input space to a lower-dimensional space.