To construct a feature space, do I have to involve all the features? In pattern recognition and machine learning, a feature vector is an n-dimensional vector of numerical features that represent some object.
The vector space associated with these vectors is often called the feature space.
Consider the titanic dataset, Pclass can take on {1, 2, 3}, sex can take on {'male','female'} ...
is it reasonable to say the feature space of Pclass is the set {1, 2, 3}?
or I have to involve all the features in the dataset, such as Pclass, set, Name, ...
 A: No, the feature space is not $\{1,2,3\}$.
You start out with observations, perhaps measuring people on height, weight, and age, trying to predict some outcome Y. Feature space is what you do with those observations. Perhaps you also want to consider the square root of height but want to exclude weight. Then your feature space is height, square root of height, and age. A linear regression would fit four parameters to this model: an intercept plus one for each of height, square root of height, and age. But you, by your decision, elect not to include weight as a feature. Whatever model you use never knows that weight was measured.
In other words, feature space is some transformation (perhaps the identity) of your data.
You can make very complicated feature spaces. That’s where you get all of the interaction terms. If you know about PCA, regressing on the PCs means that the PCs form your feature space.
(If you don’t know PCA, it’s a way of capturing much variance in few variables by taking linear combinations of the original variables.)
I love this video by MathematicalMonk (Jeffrey Miller): https://youtube.com/watch?v=rVviNyIR-fI.
