0
$\begingroup$

I am getting a little confused with the idea of composite variables. If you have a bunch of (hyptertheical) features for house price regression such as #rooms, location, proximity to school, house area and yard area. I know you could combine some of these into composite variables, such as total house area (Yard + house) or maybe some empirical classification which takes house area, number of rooms and proximity to school, applies ratings to the input values and returns a number.

I know, in some situations such variables may improve regressor performance (as mentioned by Andrew Ng in one of his excellent videos on ML Regression).

However, I assume you would then not include the original features if you use the compound features? Does it matter how the composite feature was calculated or relates the independent features? I have been trying to find some references for this problem, but I have yet to find any. Can anyone else point me in the right direction?

Thank you

BJR.

$\endgroup$
0
$\begingroup$

Other than references, I'd suggest some thought experiments to convince yourself:

However, I assume you would then not include the original features if you use the compound features?

No. Consider you generate random data from model $y=x_1+x_2+x_1x_2$, you pretend as if you don't know the actual model, but have $x_1$ and $x_2$ as your features. You decided to apply linear regression and somehow you decided to use $x_1x_2$ as an extra feature. If you remove the original features, your model assumption will be $y=ax_1x_2+b$, in which you won't be able to fit as good as the model $y=ax_1+bx_2+cx_1x_2+d$.

Does it matter how the composite feature was calculated or relates the independent features?

Yes, in the above example, imagine generating your composite feature as $x_1/x_2$ instead of $x_1x_2$. Would you still think you could catch up with the original model? Or, assume you generated a probably worse feature: $(x_1/x_2)^{0.000001}$, would you catch up as good as the above one?

A note: If you have yard area, $x_1$, house area: $x_2$, and create total area: $x_3=x_1+x_2$ as a new feature, your original model assumption $y=ax_1+bx_2+c$ doesn't really change:

$y=ax_1+bx_2+cx_3+d=ax_1+bx_2+c(x_1+x_2)+d=kx_1+lx_2+d$, i.e. still same features. (Weighted) Summing of features do not add extra capacity to linear regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.