3
$\begingroup$

I understand how linear regression is used on a sample to produce a model of how each independent variable affects the dependent variable. What I would like to do is something similar, except where the model can also represent interactions between the independent variables. Also, I would like the model to be nonlinear if possible. With preference given to simpler solutions, what is the range of available solutions to such a problem?

EDIT

To be more specific, the following are the details of my problem. I'm developing a neural network algorithm that has 11 continuous covariates that control it's behavior. These include size of different layers, learning rate, and a number of other things. I'm trying to understand how different values for the covariates produce different performance levels and why. Intuition and preliminary analysis (based on Monte Carlo sampling) tell me that there are interactions among some covariates and that effects are not likely to be linear in all cases. However, it's not obvious to me what the type of the relationships are (e.g. polynomial, exponential, etc.). Also, for efficiency, as I add more covariates and apply the algorithm to different contexts, I would like to have a method of regression in place that doesn't assume linearity and isn't dependent on foreknowledge of the type of relationship between the covariates and the performance (dependent variable).

Improve this question
$\endgroup$
9
  • $\begingroup$ @Macro only one $\endgroup$
    – Matt Munson
    Commented Jul 2, 2012 at 19:59
  • $\begingroup$ @whuber Do the details I added seem helpful? $\endgroup$
    – Matt Munson
    Commented Jul 2, 2012 at 20:05
  • $\begingroup$ Yes (+1). I removed earlier comments that are now moot in light of the edit. The additional information is revealing: for instance, it confirms my original suspicion that you're not dealing with the simple quadratic nonlinearities often assumed by statisticians whenever they see the word "interaction": yours can be as complex and mysterious as the workings of the NN itself. $\endgroup$
    – whuber ♦
    Commented Jul 2, 2012 at 20:10
  • 1
    $\begingroup$ Matt, A closer reading of the edit suggests I may be off base here: it sounds like you're not trying to peer into the NNs themselves, or replicate them through a statistical model, but only to understand how certain tuning parameters might relate to performance metrics. It would be good to be aware of Michael Chernick's distinction between nonlinear relationships between inputs and outputs and nonlinear models themselves. Even so, I still think the problem is best approached in an exploratory spirit before proceeding to more formal modeling. $\endgroup$
    – whuber ♦
    Commented Jul 2, 2012 at 20:42
  • 1
    $\begingroup$ @MattMunson There are many different types of Neural Networks, which internally use different algorithms to reduce some form of error metric. Can you say any more about the structure and algorithms that the NN in question is implementing. This might be very relevant in giving insight into how it behaves and the effect that its parameters may have on performance. $\endgroup$
    – image_doctor
    Commented Jul 3, 2012 at 7:16

1 Answer 1

Reset to default
1
$\begingroup$

The part about interaction terms in the linear model is easy to answer . If you have variables $X_1$ and $X_2$ that you think interact in the model add the term $X_1$ multiplied by $X_2$ and test to see if the coefficient is significantly different from 0. That term is your interaction term. Regarding a nonlinear model, why do you think you need a model that is nonlinear in the parameters. There are many options for nonlinear models and the choice should be dictated by how the regression parameters are connected to the covariate in a nonlinear fashion. Should the coefficients be exponents for the covariates (for example)?

Improve this answer
$\endgroup$
14
  • $\begingroup$ Michael, the term 'multivariate' in the title makes me thinks this refers to a regression model where the outcome in multivariate (since 'multiple' usually refers to the case of more than one predictor). In that case, the interaction coefficient becomes a vector and is less straightforward than what you wrote here. Some clarification is required from the OP about whether he/she really meant "multivariate". $\endgroup$
    – Macro
    Commented Jul 2, 2012 at 19:38
  • $\begingroup$ @Macro I didn't think much about the use of the term multivariate in the OPs description of the problem. Note that he mentions linear regression, independent variables but only one dependent variable. I think he is think of multiple regression rather than mutlivariate regression and just simply wants to know how interaction effects get introduced in the model. Let's see if he replies. $\endgroup$
    – Michael R. Chernick
    Commented Jul 2, 2012 at 19:49
  • 1
    $\begingroup$ @Michael There is only one dependent variable. Regarding nonlinearity, without going into the details, I am in situation where I suspect the presence of, and am not confident that I can rule out, nonlinearity. Worse though, I cannot really derive what the nature of the nonlinear relationship would be, if it is indeed nonlinear (eg. polynomial, exponential, etc.). I posted some further details above. $\endgroup$
    – Matt Munson
    Commented Jul 2, 2012 at 19:58
  • $\begingroup$ My answer is then correct for linear regression. But your terminology may still be confusing us because it sounds like you are referring to the covariates when you are using the term parameters. The variables that you think might be interacting we statisticians call covariates (or independent variables, or explanatory variables or regressors). The model coefficients are what we refer to as parameters. Linear regression require that the model be linear in the parameters and not necessarily the covariates. $\endgroup$
    – Michael R. Chernick
    Commented Jul 2, 2012 at 20:31
  • $\begingroup$ If you are concerned about nonlinearity in the covariates that does not move you into the realm of nonlinear regression. That would only be the case if you have nonlinearity in the parameters such as if you think the covariate X enters the model as a power where the power is the parameter yuo need to estimate from your data. $\endgroup$
    – Michael R. Chernick
    Commented Jul 2, 2012 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.