Non parametric estimators for noisy functions Suppose there is a function $f(a,b,c,\ldots)$ of $M$ variables (fixed numbers, not random variables). Add some Gaussian noise to this function:
$$
g(a,b,c,\ldots) = f(a,b,c,\ldots) + \varepsilon(a,b,c,\ldots)
$$
where $\varepsilon(a,b,c,\ldots) \sim N(0,\sigma_{a,b,c,\ldots}{}^2)$ are the Gaussian noise parameters. The $\sigma$ are set so that the noise is large compared to the function value, and the standard deviation of the noise depends on the input parameters (heteroskedastic).
Now suppose that I don't know $f$ or $\sigma$, but I have a large number $N$ of realisations of $g$ and each realisation has different input parameters $a,b,c,\ldots$. I am interested in estimating $f$.
If I was doing this parametrically, I could assume that $f$ is some kind of polynomial and use a regression algorithm with least squares regression. This is because the Gaussian errors "cancel out" on average because they are independent.
Is there a non-parametric approach (or semi-parametric approach) to estimate the same thing? What approach do people take in practice?
 A: Using polynomials to fit curves is a standard feature of stats 102.  However, for anything beyond the simplest and best behaved curves, polynomials are a poor choice (as you note).  A better strategy is to use cubic splines.  Your predictor space is divided into regions with a function fit to the entire space and additional functions fit to the region greater than the border (or 'knot').  Consider a case with only one $X$ variable / dimension fit with a linear spline.  Let's say the range of $X$ is partitioned at $X = .7$, then:
$$
X_{\rm spline} = \begin{cases} 0\quad    &\text{if } X\le{.7}  \\
                               X-.7\quad &\text{if } X>.7 \end{cases}
$$
Then a multiple regression model is fit using the two variables $X$ and $X_{\rm spline}$.  This is quite rudimentary, of course.  It would be more typical to have, say, 5 knots instead of one, and you can fit polynomials (typically cubics) for each variable.  This is a very powerful and flexible strategy for function approximation.  Moreover, all the benefits of linear models come naturally with this approach.  I have more information about this here: What are the advantages / disadvantages of using splines, smoothed splines, and Gaussian process emulators?
