I understand how linear regression is used on a sample to produce a model of how each independent variable affects the dependent variable. What I would like to do is something similar, except where the model can also represent interactions between the independent variables. Also, I would like the model to be nonlinear if possible. With preference given to simpler solutions, what is the range of available solutions to such a problem?
To be more specific, the following are the details of my problem. I'm developing a neural network algorithm that has 11 continuous covariates that control it's behavior. These include size of different layers, learning rate, and a number of other things. I'm trying to understand how different values for the covariates produce different performance levels and why. Intuition and preliminary analysis (based on Monte Carlo sampling) tell me that there are interactions among some covariates and that effects are not likely to be linear in all cases. However, it's not obvious to me what the type of the relationships are (e.g. polynomial, exponential, etc.). Also, for efficiency, as I add more covariates and apply the algorithm to different contexts, I would like to have a method of regression in place that doesn't assume linearity and isn't dependent on foreknowledge of the type of relationship between the covariates and the performance (dependent variable).