# Nonparametric regression and overparametrization?

I am given iid pairs $(X_i, Y_i)$, $i = 1, \ldots, n$ and wish to model the regression function $$r(x) = E[Y \mid X = x]$$ nonparametrically. Suppose I have prior knowledge that the relationship between $X$ and $Y$ is primarily a linear one, but possibly deviating from a linear relationship by an unknown function. So, $$Y_i = \beta_0 + \beta_1 X_i + g(X_i) + \epsilon_i$$ where $E[\epsilon_i] = 0$, and I wish to estimate $(\beta_0, \beta_1, g)$. But we now have a problem in that, if $g$ is unrestricted, then $(\beta_0, \beta_1, g)$ is unidentified.

My question is: how does one typically resolve this? One thought is to make some kind of orthogonality restriction like $\langle x, g\rangle = 0$ and $\langle 1, g\rangle = 0$ in the $L_2$ sense. But I have no idea how people actually do this in practice. Is it feasible to do something like minimize $$Q(\beta, g) = \sum_i \left\{Y_i - \beta_0 - \beta_1 X_i - g(X_i)\right\}^2 \\ - c \int (g'')^2 \ dx - \lambda_1 \int g \ dx - \lambda_2 \int gx \ dx,$$ where $c$ is a fixed smoothness penalty and the $\lambda$s are Lagrange multipliers to enforce the conditions above? Or is my way of going about things with orthogonality constraints the wrong way of thinking about things?

The number of parameters (n+3, including the unknown error variance) exceeds the size of the sample (that's the "incidental parameters" situation). The two orthogonality restrictions that you mention reduce the number of unknowns by 2, no more.

But if you are interested mainly in estimating the $\beta$'s, then these restrictions permit you (through a method-of-moments approach) to obtain unbiased and consistent estimates of the two betas by standard OLS. This has general validity -when a regressor is assumed orthogonal to the others in a multiple regression setting, it can "leave the scene" without affecting the least-square estimates of the other parameters.
The two orthogonality restrictions in moment notation are

$$E(X_ig(X_i))=0,\;\; E(g(X_i))=0\;\; \forall i$$

The method of moments approach dictates to obtain estimates by estimating the sample analogues of these restrictions. These sample analogues are

$$\frac 1n \sum_{i=1}^nx_ig(x_i)=0,\;\; \frac 1n \sum_{i=1}^ng(X_i)=0$$

From the regression equation we have

$$y_i - \beta_0 - \beta_1 X_i - \epsilon_i = g(x_i) ,\;\; \forall i$$ Substituting we obtain

$$\frac 1n \sum_{i=1}^nx_i\Big(y_i - \beta_0 - \beta_1 x_i - \epsilon_i\Big)=0\\ \frac 1n \sum_{i=1}^n\Big(y_i - \beta_0 - \beta_1 x_i - \epsilon_i\Big)=0$$

The unknown error term is ignored. Carrying out the multiplications we obtain

$$\frac 1n \sum_{i=1}^n\Big(x_iy_i - \beta_0x_i - \beta_1 x_i^2 \Big)=0\\ \frac 1n \sum_{i=1}^n\Big(y_i - \beta_0 - \beta_1 x_i \Big)=0$$

which lead us to the usual OLS estimators in a simple regression setting (the bar denoting the sample mean)

$$\hat \beta_1 = \frac {\frac 1n \sum_{i=1}^nx_iy_i - \bar y\bar x}{\frac 1n \sum_{i=1}^nx_i^2 - \bar x}, \;\; \hat \beta_0 = \bar y -\hat \beta_1\bar x$$

• but this doesn't allow to derive $g$, right? although topicstarter's smoothness condition seems should make $g$ identifiable. basically, he could just smooth residuals from OLS with some kernel smoothing and this should be $g$ Dec 11 '13 at 1:47
• @Kochede Indeed. In other words, we need additional assumptions to estimate the $g-series$ point per point. ("smoothing" essentially implies some assumptions). Dec 11 '13 at 1:57
• But topicstarter already has this additional assumption - smoothness penalty term in his optimization $−c\int(g′′)^2 dx$. When you say that $g$ is unidentifiable, this is true only if you ignore this "smoothness" restriction of class of functions. Dec 11 '13 at 2:42
• @Kochede I don't see where we disagree. My answer was meant to show that the orthogonality conditions alone permit us to identify and estimate the betas. Then, as you point out, we can obtain in a second step estimates for the $g$'s, by smoothing the OLS residuals. Dec 11 '13 at 2:47
• Ah, I see :) I thought you didn't notice smoothness constraint. Dec 11 '13 at 2:55

You could replace your optimization penalties with their sample equivalents and minimize something like: $$Q(\beta, g) = \sum_i \left\{Y_i - \beta_0 - \beta_1 X_i - g(X_i)\right\}^2 \\ - c \sum_i (\hat g_i'')^2 - \lambda_1 \sum_i g_i - \lambda_2 \sum_i g_i X_i ,$$

Here $g_i = g(X_i)$, $\hat g_i'= (g_{i+1}-g_i)/(X_{i+1}-X_i)$ and $\hat g_i''= (\hat g_{i+1}'-\hat g_i')/(X_{i+1}-X_i)$. Not sure about particular way to approximate derivative with differences - you may think of a better way.

Alternatively, you could estimate $\beta$'s with methods of moments as @Alecos suggested and then just kernel-smooth residuals from there to get an estimate of $g$.