# Is non-linear least squares the best approximation of the corresponding population conditional expectation in a least squared error sense?

Background

Loosely adapting Angrist & Pischke Mostly Harmless Econometrics (A&P) section 3.1, suppose we have a conditional expectation function (CEF) $$\mathbb{E}\left[Y_i|X_i\right]$$ defined such that $$\mathbb{E}\left[Y_i|X_i=x\right]\equiv\int t f_y\left(t|X_i=x\right)dt$$ A&P prove that given the OLS problem (3.1.2) $$\beta \equiv argmin_b \mathbb{E}\left[\left(Y_i-X_i' b\right)^2\right]$$ $$X_i'\beta$$ provides the minimum least squared error to the CEF, that is (3.1.4) $$\beta = argmin_b \mathbb{E}\left\{\left(\mathbb{E}\left[Y_i|X_i\right]-X_i'b\right)^2\right\}$$ I won't walk through their proof here since I'm basically going to follow it as part of the question below.

Question

Can the above be adapted to motivate the use of non-linear least squares?

Here is my attempt. Consider the below non-linear least-squares problem $$\vartheta\equiv argmin_\theta \mathbb{E}\left[\left(Y_i-g(X_i,\theta)\right)^2\right]$$ where $$g(X_i,θ)$$ is assumed continuous but not necessarily differentiable.

Then adapting A&P 3.1.4 and its corresponding proof, it is also true that $$g(X_i,\vartheta)$$ provides the minimum least-squares approximation to $$\mathbb{E}\left[Y_i|X_i\right]$$ among all parameter choices, that is, $$\vartheta = argmin_\theta \mathbb{E}\left[\left(\mathbb{E}\left[Y_i|X_i\right]-g(X_i,\theta)\right)^2\right]$$ This is true because \begin{aligned} \left(Y_i-g(X_i,\theta)\right)^2 &= \left[\left(Y_i-\mathbb{E}\left[Y_i|X_i\right]\right)+\left(\mathbb{E}\left[Y_i|X_i\right]-g(X_i,\theta)\right)\right]^2 \\ &=\left(Y_i-\mathbb{E}\left[Y_i|X_i\right]\right)^2\\ &+\left(\mathbb{E}\left[Y_i|X_i\right]-g(X_i,\theta)\right)^2\\ &+\left(Y_i-\mathbb{E}\left[Y_i|X_i\right]\right)\left(\mathbb{E}\left[Y_i|X_i\right]-g(X_i,\theta)\right) \end{aligned} The first term of the final expansion is not a function of the parameters $$\theta$$. The last term is zero by iterated expectations \begin{aligned} &\mathbb{E}\left\{\left(Y_i-\mathbb{E}\left[Y_i|X_i\right]\right)\left(\mathbb{E}\left[Y_i|X_i\right]-g(X_i,\theta)\right)\right\}\\ &= \mathbb{E}\left\{\left(\mathbb{E}\left[Y_i|X_i\right]-\mathbb{E}\left[Y_i|X_i\right]\right)\left(\mathbb{E}\left[Y_i|X_i\right]-g(X_i,\theta)\right)\right\}\\ &=0 \end{aligned}

Is the above reasoning correct? Does it motivate non-linear least squares?

Also would be great to have a citation, but any help is appreciated.