Composite Variables

Question

I am getting a little confused with the idea of composite variables. If you have a bunch of (hyptertheical) features for house price regression such as #rooms, location, proximity to school, house area and yard area. I know you could combine some of these into composite variables, such as total house area (Yard + house) or maybe some empirical classification which takes house area, number of rooms and proximity to school, applies ratings to the input values and returns a number.

I know, in some situations such variables may improve regressor performance (as mentioned by Andrew Ng in one of his excellent videos on ML Regression).

However, I assume you would then not include the original features if you use the compound features? Does it matter how the composite feature was calculated or relates the independent features? I have been trying to find some references for this problem, but I have yet to find any. Can anyone else point me in the right direction?

Thank you

BJR.

Answer 1

Other than references, I'd suggest some thought experiments to convince yourself:

However, I assume you would then not include the original features if you use the compound features?

No. Consider you generate random data from model $y=x_1+x_2+x_1x_2$, you pretend as if you don't know the actual model, but have $x_1$ and $x_2$ as your features. You decided to apply linear regression and somehow you decided to use $x_1x_2$ as an extra feature. If you remove the original features, your model assumption will be $y=ax_1x_2+b$, in which you won't be able to fit as good as the model $y=ax_1+bx_2+cx_1x_2+d$.

Does it matter how the composite feature was calculated or relates the independent features?

Yes, in the above example, imagine generating your composite feature as $x_1/x_2$ instead of $x_1x_2$. Would you still think you could catch up with the original model? Or, assume you generated a probably worse feature: $(x_1/x_2)^{0.000001}$, would you catch up as good as the above one?

A note: If you have yard area, $x_1$, house area: $x_2$, and create total area: $x_3=x_1+x_2$ as a new feature, your original model assumption $y=ax_1+bx_2+c$ doesn't really change:

$y=ax_1+bx_2+cx_3+d=ax_1+bx_2+c(x_1+x_2)+d=kx_1+lx_2+d$, i.e. still same features. (Weighted) Summing of features do not add extra capacity to linear regression.