Consider the following model:
$$ y_{i}=f\left(\boldsymbol{x}_{i};\theta\right)+\varepsilon_{i} $$
where $y_{i}$ is the dependent variable, $\boldsymbol{x}_{i}$ is a vector of explanatory variables, $\varepsilon_{i}$ is $iid\left(0,\sigma^{2}\right)$ noise (here presented as additive noise, but it could be more general) and $\theta$ is a vector of unknown parameters.
$f$ is a known continuous differentiable function.The obvious choice would be to use non-linear regression to estimate the unknown parameters $\theta$. But lets say I insist on using linear regression which have an analytical solution. What choices do I have? I am not interested in doing inference on the parameters $\theta$. I am more interested in an approximation:
$$f\left(\boldsymbol{x}_{i};\theta\right)\approx g\left(\boldsymbol{x}_{i}\right)\beta $$
linear in the parameters $\beta$.
One could specify $g$ as being polynomials of $\boldsymbol{x}_{i}$, to approximate a broad range of functions. However, this approach will not make use of any information about the known function $f$. For other functional forms e.g. $f\left(x\right)=e^{x\theta}$ one may use a transformation of the $y$-variable, but this is not what I am looking for either.
