How to estimate a multiple regression function that has interactions between the independent variables and is potentially non-linear?

I understand how linear regression is used on a sample to produce a model of how each independent variable affects the dependent variable. What I would like to do is something similar, except where the model can also represent interactions between the independent variables. Also, I would like the model to be nonlinear if possible. With preference given to simpler solutions, what is the range of available solutions to such a problem?

EDIT

To be more specific, the following are the details of my problem. I'm developing a neural network algorithm that has 11 continuous covariates that control it's behavior. These include size of different layers, learning rate, and a number of other things. I'm trying to understand how different values for the covariates produce different performance levels and why. Intuition and preliminary analysis (based on Monte Carlo sampling) tell me that there are interactions among some covariates and that effects are not likely to be linear in all cases. However, it's not obvious to me what the type of the relationships are (e.g. polynomial, exponential, etc.). Also, for efficiency, as I add more covariates and apply the algorithm to different contexts, I would like to have a method of regression in place that doesn't assume linearity and isn't dependent on foreknowledge of the type of relationship between the covariates and the performance (dependent variable).

• @Macro only one – Matt Munson Jul 2 '12 at 19:59
• @whuber Do the details I added seem helpful? – Matt Munson Jul 2 '12 at 20:05
• Yes (+1). I removed earlier comments that are now moot in light of the edit. The additional information is revealing: for instance, it confirms my original suspicion that you're not dealing with the simple quadratic nonlinearities often assumed by statisticians whenever they see the word "interaction": yours can be as complex and mysterious as the workings of the NN itself. – whuber Jul 2 '12 at 20:10
• Matt, A closer reading of the edit suggests I may be off base here: it sounds like you're not trying to peer into the NNs themselves, or replicate them through a statistical model, but only to understand how certain tuning parameters might relate to performance metrics. It would be good to be aware of Michael Chernick's distinction between nonlinear relationships between inputs and outputs and nonlinear models themselves. Even so, I still think the problem is best approached in an exploratory spirit before proceeding to more formal modeling. – whuber Jul 2 '12 at 20:42
• @MattMunson There are many different types of Neural Networks, which internally use different algorithms to reduce some form of error metric. Can you say any more about the structure and algorithms that the NN in question is implementing. This might be very relevant in giving insight into how it behaves and the effect that its parameters may have on performance. – image_doctor Jul 3 '12 at 7:16

The part about interaction terms in the linear model is easy to answer . If you have variables $X_1$ and $X_2$ that you think interact in the model add the term $X_1$ multiplied by $X_2$ and test to see if the coefficient is significantly different from 0. That term is your interaction term. Regarding a nonlinear model, why do you think you need a model that is nonlinear in the parameters. There are many options for nonlinear models and the choice should be dictated by how the regression parameters are connected to the covariate in a nonlinear fashion. Should the coefficients be exponents for the covariates (for example)?