What is "feature space"?

Question

What is the definition of "feature space"?

For example, When reading about SVMs, I read about "mapping to feature space". When reading about CART, I read about "partitioning to feature space".

I understand what's going on, especially for CART, but I think that there is some definition that I have missed.

Is there a general definition of "feature space"?

Is there a definition that will give me more insight into SVM kernels and/or CART?

Answer 1

Feature Space

Feature space refers to the $n$-dimensions where your variables live (not including a target variable, if it is present). The term is used often in ML literature because a task in ML is feature extraction, hence we view all variables as features. For example, consider the data set with:

Target

1. $Y \equiv$ Thickness of car tires after some testing period

Variables

1. $X_1 \equiv$ distance travelled in test
2. $X_2 \equiv$ time duration of test
3. $X_3 \equiv$ amount of chemical $C$ in tires

The feature space is $\mathbf{R}^3$, or more accurately, the positive quadrant in $\mathbf{R}^3$ as all the $X$ variables can only be positive quantities. Domain knowledge about tires might suggest that the speed the vehicle was moving at is important, hence we generate another variable, $X_4$ (this is the feature extraction part):

• $X_4 =\frac{X_1}{X_2} \equiv$ the speed of the vehicle during testing.

This extends our old feature space into a new one, the positive part of $\mathbf{R}^4$.

Mappings

Furthermore, a mapping in our example is a function, $\phi$, from $\mathbf{R}^3$ to $\mathbf{R}^4$:

$$\phi(x_1,x_2,x_3) = (x_1, x_2, x_3, \frac{x_1}{x_2} )$$