I want add to my regression additional features to complicate my model and lower bias, after searching around internet is seems a good idea is to add square features, that can help regression learn in coefficients more knowledge.

I want add to my regression model square of pairs of features $x * y$, i think that can help learn my model relationships between features variables that can be highly correlated.

Problem is when I see at interpretation of $x * y$ it seems incorrect, when i get $high * high = higher$ ex. $5 * 5 = 25$ that is good, but $low * low = higher$ seems strongly invalid ex. $-5 * -5 = 25$, should be $-25$ for my problem statement.

Same problems I have for $high * low = low$, ex. $5 * -5 = -25$, i think should be somewhere between $<-5; 5>$, for example in $0$.

That anyone have any thoughts about what equation I can use to get correct representation?

I will appreciate other ideas for new virtual calculated features.

  • $\begingroup$ It's definitely common to add features given by basis functions as you've described, as it allows the modeling of arbitrary nonlinearities in the input space - however, it's not obvious what sort of nonlinearities you're trying to capture; if we know that, perhaps we can give you more insight. What was your reason for choosing the product of x and y? What do those quantities represent? $\endgroup$ – Louis Cialdella Dec 9 '13 at 22:21
  • $\begingroup$ @LCialdella: I just read about square of pairs, it in my case making RMSE lower but slightly, i think if representation will be correct it will be work better, I try to capture what pair of features predicting good when used combine, some of feature pairs are highly correlated, i think that regression can learn to score more feature pairs that are not strongly correlated but i need function that will be have correct representation, when 2 highs are combined that should be higher, when 2 low values then should be lower not higher $\endgroup$ – Svisstack Dec 9 '13 at 22:47
  • $\begingroup$ @LCialdella: to simplify i just want add more variance in my regression, because some of my features are highly correlated, i dont know how to make equation to catch that. $\endgroup$ – Svisstack Dec 9 '13 at 22:49
  • $\begingroup$ This question is difficult to follow. Can you revise the wording to better fit standard English? Beyond that, many textbooks on regression discuss ways to include nonlinear terms and interaction terms; is there some specific guidance you've encountered that you want to understand better? Merely to ask for suggestions about how to build a more comprehensive or nuanced regression model makes this question too broad for this site, I'm afraid. $\endgroup$ – rolando2 Dec 10 '13 at 5:01
  • $\begingroup$ @Rolando2: better fit - mean lower RSME on testing test, when I asking that question i think that is some common problem, now I see I'm on my own, because problem is strongly case dependent. $\endgroup$ – Svisstack Dec 10 '13 at 11:41

I don't see this as a problem for linear regression.
If I understand you correctly then if all your data is between +/-5 then x_new=(x+10) y_new=(y+10) would solve your problems. (ie that you want low * low = low, low * high = medium etc)

but adding a constant to your inputs doesn't change the potential models in any way:

w_new_xy * x_new *y_new = w_new_xy *x *y + w_new * 10 * x + w_new * 10 * y + w_new * 100

in other words your existing model also handles this ( by having different weights for the linear terms and bias)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.