# To construct a feature space, do I have to involve all the features?

In pattern recognition and machine learning, a feature vector is an n-dimensional vector of numerical features that represent some object.

The vector space associated with these vectors is often called the feature space.

Consider the titanic dataset, Pclass can take on {1, 2, 3}, sex can take on {'male','female'} ...

is it reasonable to say the feature space of Pclass is the set {1, 2, 3}?

or I have to involve all the features in the dataset, such as Pclass, set, Name, ...

No, the feature space is not $$\{1,2,3\}$$.

You start out with observations, perhaps measuring people on height, weight, and age, trying to predict some outcome Y. Feature space is what you do with those observations. Perhaps you also want to consider the square root of height but want to exclude weight. Then your feature space is height, square root of height, and age. A linear regression would fit four parameters to this model: an intercept plus one for each of height, square root of height, and age. But you, by your decision, elect not to include weight as a feature. Whatever model you use never knows that weight was measured.

In other words, feature space is some transformation (perhaps the identity) of your data.

You can make very complicated feature spaces. That’s where you get all of the interaction terms. If you know about PCA, regressing on the PCs means that the PCs form your feature space.

(If you don’t know PCA, it’s a way of capturing much variance in few variables by taking linear combinations of the original variables.)

I love this video by MathematicalMonk (Jeffrey Miller): https://youtube.com/watch?v=rVviNyIR-fI.

• Thanks for your reply. What if I consider a reduced dataset which only contains 2 features "Pclass" and "sex"? what the feature space is now? Nov 13, 2019 at 21:34
• @JJJohn it still depends on what you want in your feature space. You may just want those two variables. Perhaps you want an interaction term, too.
– Dave
Nov 13, 2019 at 23:51
• Thanks for your kindness. Actually, "what I want" is the question. in other words, "what do I need" in feature space to train a classifier that could be used to predict "Survived", given a dataset composed of 2 features "Pclass", "sex" and 1 label "Survived", what the feature space is in this case. Nov 14, 2019 at 0:17
• @JJJohn it still depends on you and the decisions you make about what you want to include in your model. Maybe you just want those two variables. Maybe you want their interaction, too. Maybe you want to exclude one. The decision is yours. Did you watch Monk’s video, yet? As far as what you need in your model, that is part of your job as a statistician or data scientist: to figure that out!
– Dave
Nov 14, 2019 at 0:22