Composite Variables

I am getting a little confused with the idea of composite variables. If you have a bunch of (hyptertheical) features for house price regression such as #rooms, location, proximity to school, house area and yard area. I know you could combine some of these into composite variables, such as total house area (Yard + house) or maybe some empirical classification which takes house area, number of rooms and proximity to school, applies ratings to the input values and returns a number.

I know, in some situations such variables may improve regressor performance (as mentioned by Andrew Ng in one of his excellent videos on ML Regression).

However, I assume you would then not include the original features if you use the compound features? Does it matter how the composite feature was calculated or relates the independent features? I have been trying to find some references for this problem, but I have yet to find any. Can anyone else point me in the right direction?

Thank you

BJR.

1 Answer

Other than references, I'd suggest some thought experiments to convince yourself:

However, I assume you would then not include the original features if you use the compound features?

No. Consider you generate random data from model $$y=x_1+x_2+x_1x_2$$, you pretend as if you don't know the actual model, but have $$x_1$$ and $$x_2$$ as your features. You decided to apply linear regression and somehow you decided to use $$x_1x_2$$ as an extra feature. If you remove the original features, your model assumption will be $$y=ax_1x_2+b$$, in which you won't be able to fit as good as the model $$y=ax_1+bx_2+cx_1x_2+d$$.

Does it matter how the composite feature was calculated or relates the independent features?

Yes, in the above example, imagine generating your composite feature as $$x_1/x_2$$ instead of $$x_1x_2$$. Would you still think you could catch up with the original model? Or, assume you generated a probably worse feature: $$(x_1/x_2)^{0.000001}$$, would you catch up as good as the above one?

A note: If you have yard area, $$x_1$$, house area: $$x_2$$, and create total area: $$x_3=x_1+x_2$$ as a new feature, your original model assumption $$y=ax_1+bx_2+c$$ doesn't really change:

$$y=ax_1+bx_2+cx_3+d=ax_1+bx_2+c(x_1+x_2)+d=kx_1+lx_2+d$$, i.e. still same features. (Weighted) Summing of features do not add extra capacity to linear regression.