2
$\begingroup$

Suppose we have a function, $$ F = F(x_1, x_2, ... , x_N) , $$ then one can try to model it with one of the many available techniques, for example, via linear regression: $$ F = \sum_{i=1} ^N c_i x_i $$ But linear regression, as well as any other technique I am aware of assumes that the variables are independent of each other. The question is: how can I model a function with variables depending on each other, i.e. $$ F = F \left[y_1 (x_1, x_2, ... , x_N), y_2 (x_1, x_2, ... , x_N), ... , y_M (x_1, x_2, ... , x_N), x_1, x_2, ... , x_N\right] ? $$

$\endgroup$
2
  • 1
    $\begingroup$ Why do you think that the predictor variables need to be independent of each other? In practice, predictors are often correlated; there is an extensive set of questions with a multicollinearity tag on this site, dealing with this situation. $\endgroup$
    – EdM
    Jun 26, 2017 at 18:37
  • $\begingroup$ This question might be difficult to answer, because it is predicated on a false assumption: almost no regression method assumes "independence" of the independent variables (in any sense of "independent", of which there are several). Perhaps that information alone settles the question--but if not, could you describe the particular problem you are addressing and explain why it leads you to suppose that standard methods do not apply? $\endgroup$
    – whuber
    Jun 26, 2017 at 19:19

1 Answer 1

1
$\begingroup$

But linear regression, as well as any other technique I am aware of assumes that the variables are independent of each other.

If by "variables" you mean the "features", then this is not correct for linear regression or most of the other common supervised learning models. Features can be, and often are, deterministic functions of each other, such as using polynomials.

Indeed, what you propose is completely routine. Perhaps the easiest example is a model with an interaction between two features: $\hat y = \hat\beta_0 + \hat\beta_1x_1 + \hat\beta_2x_2 + \hat\beta_3x_1x_2$.

In that regression, you have the original two features, $x_1$ and $x_2$. However, you also have a function (often called a "basis function") of both features to give the $x_1x_2$ term.

You can extend this idea to as many features and basis functions as you want. For instance, the following is a totally legitimate linear regression.

$$ \hat y = \hat\beta_0 + \hat\beta_1x_1 + \hat\beta_2x_2 + \hat\beta_3x_3 + + \hat\beta_4\cos(x_1x_2) + \hat\beta_5x_1^{x_2^{x_3}} $$

Using your notation, the $y_i$ would be the basis functions of the original features.

$\endgroup$
1
  • $\begingroup$ I've liked Jeffrey Miller's (MathematicalMonk on YouTube) explanation of basis functions since I first saw it. $\endgroup$
    – Dave
    Jun 2, 2023 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.