How can I build a (regression) model with 'independent' variables depending on each other

Question

Suppose we have a function, $$ F = F(x_1, x_2, ... , x_N) , $$ then one can try to model it with one of the many available techniques, for example, via linear regression: $$ F = \sum_{i=1} ^N c_i x_i $$ But linear regression, as well as any other technique I am aware of assumes that the variables are independent of each other. The question is: how can I model a function with variables depending on each other, i.e. $$ F = F \left[y_1 (x_1, x_2, ... , x_N), y_2 (x_1, x_2, ... , x_N), ... , y_M (x_1, x_2, ... , x_N), x_1, x_2, ... , x_N\right] ? $$

Answer 1

But linear regression, as well as any other technique I am aware of assumes that the variables are independent of each other.

If by "variables" you mean the "features", then this is not correct for linear regression or most of the other common supervised learning models. Features can be, and often are, deterministic functions of each other, such as using polynomials.

Indeed, what you propose is completely routine. Perhaps the easiest example is a model with an interaction between two features: $\hat y = \hat\beta_0 + \hat\beta_1x_1 + \hat\beta_2x_2 + \hat\beta_3x_1x_2$.

In that regression, you have the original two features, $x_1$ and $x_2$. However, you also have a function (often called a "basis function") of both features to give the $x_1x_2$ term.

You can extend this idea to as many features and basis functions as you want. For instance, the following is a totally legitimate linear regression.

$$ \hat y = \hat\beta_0 + \hat\beta_1x_1 + \hat\beta_2x_2 + \hat\beta_3x_3 + + \hat\beta_4\cos(x_1x_2) + \hat\beta_5x_1^{x_2^{x_3}} $$

Using your notation, the $y_i$ would be the basis functions of the original features.