When I use regression models I feel leery of defaulting to an assumptions of linear association; instead I like to explore the functional form of relationships between dependent and explanatory variables using nonparametric smoothing regression (e.g. generalized additive models, lowess/lowess, running line smoothers, etc.) before estimating a parametric model using, as appropriate, nonlinear least squares regression to estimate parameters for functions suggested by the nonparametric model.

What is a good way to think about performing cross validation in the nonparametric smoothing regression phase of such an approach? I wonder if I might encounter a situation where in random holdout sample A a relationship approximated by a "broken stick" linear hinge function might be evident, while holdout sample B suggests a relationship that would be better approximated by a parabolic threshold hinge function.

Would one take a non-exhaustive approach hold back some randomly selected portion of the data, perform the nonparametric regression, interpret plausible functional forms for the result, and repeat this a few (human-manageable) number of times and mentally tally plausible functional forms?

Or would one take an exhaustive approach (e.g. LOOCV), and use some algorithm to 'smooth all the smooths' and used that smoothest of smooths to inform plausible functional forms? (Although, on reflection, I think LOOCV is quite unlikely to result in very different functional relationships since a functional form on a large enough sample is unlikely to be altered by a single data point.)

My applications will typically entail human-manageable numbers of predictor variables (a handful to a few dozen, say), but my sample sizes will range from from a few hundreds to a few hundred thousand. My aim is to produce an intuitively communicated and easily translated model that might be used to make predictions by people with data sets other than mine, and which do not include the outcome variables.

References in answers very welcome.

  • $\begingroup$ It would help to clarify your question: what do you want to achieve with cross-validation? Determine which smoothing model works best? $\endgroup$ – jubo Aug 25 '14 at 3:24
  • $\begingroup$ Thank you. I am interested in generating cross-validated parametric predictive models. However, I am not comfortable with the assumption of linear functional relationships between dependent and independent variables. Hence, the nonparametric -> parametric nonlinear approach I described above. I'm interested in getting CV in on the first step in order to validate the suggested functional form(s) (e.g. do different held-out sub-samples suggest different function?). $\endgroup$ – Alexis Aug 25 '14 at 7:08

It seems to me there are two confusions in your question:

  • First, linear (least-square) regression does not require a linear relationship in the independent variables, but in the parameters.

    Thus $y=a + b \cdot x e^{-x} + c \cdot \frac{z}{1 + x^2}$ can be estimated by ordinary least squares ($y$ is a linear function of parameters $a$, $b$, $c$), while $y = a + b \cdot x + b^2 \cdot z$ cannot ($y$ is not linear in parameter $b$).

  • Second, how do you determine a "correct" functional model from a smoother, i.e. how do you go from step 1 to step 2?

    As far as I know, there is no way to infer "which functions of regressors to use" from smoothing techniques such as splines, neural nets, etc. Except maybe by plotting the smoothed outputs, and determining relationships by intuition, but that doesn't sound very robust to me, and it seems one doesn't need smoothing for this, just scatterplots.

If your final goal is a linear regression model, and your problem is that you don't know exactly what functional form of the regressors should be used, you would be better off directly fitting a regularized linear regression model (such as LASSO) with a large basis expansion of the original regressors (such as polynomials of the regressors, exponentials, logs, ...). The regularization procedure should then elimnate the unneeded regressors, leaving you with a (hopefully good) parametric model. And you can use cross-validation to determine the optimal penalization parameter (which determines the actual degrees of freedom of the model).

You can always use nonparametric regressions as a benchmark for generalization error, as a way to check that your regularized linear model predicts outside data just as well as a nonparametric smoother.

  • 1
    $\begingroup$ Your 1st point: I am communicating poorly. I am interested in the nonlinear parameterizations as you indicate (e.g. the hinge function: $\beta_{x}x + \beta_{\text{h}}\max(x-\theta,0)$, where $\theta$ is also a parameter to be estimated). The motivation for this question is because I wonder: will hold-out sub-sample A suggest, say, a "broken stick" hinge function, while hold-out sub-sample B suggests a parabolic threshold hinge function? I am not looking for "correct" (possibly nonlinear in the parameterizations) functional relationships, but "predictive" ones. Does CV has a role to play here? $\endgroup$ – Alexis Aug 25 '14 at 18:48
  • $\begingroup$ Your 2nd point: You are correct to say that moving from step 1 to 2 entails intuition. However, one benefit of using such a "not-robust" approach (as opposed to algorithmic curve fitting approaches), is (in my experience) one can communicate the relationship between outcome and predictor with a reasonably intuitive parameterization (e.g. fractional polynomial functions which do not call a particular shape/form to mind), with errors that are reasonably close to the algorithmic approach. So: better than linear parameter assumptions, but retaining interpretability for the audience. $\endgroup$ – Alexis Aug 25 '14 at 18:58
  • $\begingroup$ It is not clear to me that LASSO would serve my needs: I am not in a $p > n$ situation. $\endgroup$ – Alexis Aug 27 '14 at 14:41
  • 1
    $\begingroup$ I'd urge you to reconsider your lasso comment. Although lasso was designed with $p > n$ in mind, its usefulness is not limited to that situation, especially when seeking a parsimonious (small) model with high predictive power. jubo was suggesting expanding the number of covariates by adding a large basis expansion, splines or orthogonal polynomials, and then letting lasso choose the relevant ones. $\endgroup$ – Matthew Drury May 13 '15 at 4:15
  • $\begingroup$ I'm not sure a lasso approach to splines is really ideal though, because they tend to have narrow support (for cubics I believe they are supported between four consecutive knots). Some combination of L1 and L2 penalties with glmnet may work though. $\endgroup$ – Matthew Drury May 13 '15 at 4:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.