Suppose I have a simple linear model with two variables and their interactions:

$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 x_2 + \beta_4 x_1^2 + \beta_5 x_2^2 + \epsilon$

where the $\beta_i$ can, of course, be estimated by OLS. Assume that the standard Gauss-Markov assumptions of OLS hold, and that data points are sampled i.i.d. My question is the following: Is there a way to approximate this model such that the approximation is separable in $x_1, x_2$?

Clearly, I could just omit the term $\beta_3 x_1 x_2$ and obtain a separable model at the cost of accuracy, but I'm wondering if there is an approach which would better approximate the original model? I'm only interested in predicting $y$, so the metric by which an approximation would be assessed would be the additional error in introduces in predicting $y$. In other words, I'm interested in an approximation which minimizes the difference in sum-squared-residuals relative to OLS.

The approximation need not be linear in its parameters (or even parametric). This seems related to the idea behind partitioned regression. I also know that orthogonal polynomials are commonly applied to function approximation, but again, am not sure if it would be useful here.

If anyone has any suggestions about where to look for some ideas/theory I'd be happy to hear them.

  • 2
    $\begingroup$ Your problem will not be sufficiently well defined until you can provide a quantitative way to evaluate how well any approximation works. How are we supposed to measure the goodness of an approximation? The usual methods based on likelihoods or sums of squares are not available because there are no data in evidence. $\endgroup$ – whuber Sep 18 '17 at 21:59
  • $\begingroup$ Thanks, I've edited my post to clarify the meaning of a "good" approximation. $\endgroup$ – aht Sep 18 '17 at 23:45
  • $\begingroup$ I don't follow this. Why don't you just include the interaction term & then use OLS? That will do exactly what you want. What, then, is your question? $\endgroup$ – gung - Reinstate Monica Sep 19 '17 at 0:55
  • $\begingroup$ @gung - there are sometimes computational advantages to having a separable model. If the cost (in terms of accuracy) incurred by using an approximation is not too high then it may be an acceptable price to pay for computational efficiency. $\endgroup$ – aht Sep 19 '17 at 2:05

If you really need separability for some reason, you can often get a reasonable approximation by transformation of both response and predictors; however the resulting relationships - while still smooth - will generally require additive modeling.

You might, for example look into approaches like ACE (alternating conditional expectations -- see Breiman, L. and Friedman, J. H. (1985). "Estimating optimal transformations for multiple regression and correlation.", JASA 80 , 580-78. This is implemented as ace in package acepack for R), for example.

You can do simpler things; if you fit reasonably flexible models in the two predictors (such as via splines for example), you can use Box-Cox transformations (and even estimate the $\lambda$ parameter) to sometimes get reasonable transformations for the response. While less flexible this may be easier to interpret and explain.

There are a wide variety of other approaches to getting approximate transformations for additivity -- indeed it's essentially the same notion as Lafay anamorphosis in nomography which dates back well over a century.

  • $\begingroup$ Thanks @Glen_b - this sounds like a promising approach. A quick look into ACE makes it seem similar to GAMs which had also seemed like a possible route to take. As to this statement: "There are a wide variety of other approaches to getting approximate transformations for additivity" - can you suggest any good surveys of the topic? $\endgroup$ – aht Sep 19 '17 at 2:03
  • $\begingroup$ @aht I don't presently recall any surveys of the topic as a general thing; a bunch of different approaches touch on the topic in some way. GAMs won't generally help you if you don't transform the y (the interaction is what makes it not additive, a more flexible form of additivity doesn't get you far if you don't change how far from additive the relationship is); once you do something to make the relationship close to additive, GAMs should do the rest of the job just fine. $\endgroup$ – Glen_b Sep 19 '17 at 2:14
  • $\begingroup$ As a simple example consider a relationship (without error) like $y = 9x_1^2 + 6 x_1x_2+x_2^2 + 12x_1+4x_2+8$. That $x_1x_2$ term makes it not-additive; if you model it as $y=f(x_1)+g(x_2)$ you'll have a very distinct pattern in your residuals (a sort of saddle shape). But if you look at $\sqrt{y-4}$, in this case you get a nice linear function in $x_1$ and $x_2$ and zero error. In general no transformation will induce actual additivity of the predictors but you can often get a quite reasonable approximation. $\endgroup$ – Glen_b Sep 19 '17 at 2:25
  • $\begingroup$ The introduction of this paper on ACE does mention some of the approaches and in passing mentions the distinction between non-transformative approaches like GAM and methods that transform (like ACE) $\endgroup$ – Glen_b Sep 19 '17 at 4:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.