# Non parametric estimators for noisy functions

Suppose there is a function $f(a,b,c,\ldots)$ of $M$ variables (fixed numbers, not random variables). Add some Gaussian noise to this function:

$$g(a,b,c,\ldots) = f(a,b,c,\ldots) + \varepsilon(a,b,c,\ldots)$$

where $\varepsilon(a,b,c,\ldots) \sim N(0,\sigma_{a,b,c,\ldots}{}^2)$ are the Gaussian noise parameters. The $\sigma$ are set so that the noise is large compared to the function value, and the standard deviation of the noise depends on the input parameters (heteroskedastic).

Now suppose that I don't know $f$ or $\sigma$, but I have a large number $N$ of realisations of $g$ and each realisation has different input parameters $a,b,c,\ldots$. I am interested in estimating $f$.

If I was doing this parametrically, I could assume that $f$ is some kind of polynomial and use a regression algorithm with least squares regression. This is because the Gaussian errors "cancel out" on average because they are independent.

Is there a non-parametric approach (or semi-parametric approach) to estimate the same thing? What approach do people take in practice?

• I'm not sure that including a polynomial makes something parametric. I usually think of parametric as having parameters of a known distribution, such as the normal, and nonparametrics don't assume such a distribution, perhaps using ranks. What do you mean by "non-parametric" here? Would quantile regression or ordinal logistic regression (both of which could have polynomial expansions of X) count? Why do you want non-parametric? Are you just worried about the heteroscedasticity? Commented Dec 15, 2014 at 22:43
• Fair point. In my field, "parametric" is taken to mean that you define a model (in this case a polynomial) and fit the data to that model. I would define something like interpolation to be non-parametric because no model is used. I realise that this probably isn't self consistent. Commented Dec 15, 2014 at 22:54
• Regarding the rest of your comment, the important point is that under the assumption of independent normal residuals least squares regression "cancels out" the noise. Would the same hold true for quantile regression/ordinal logisitc regression? Commented Dec 15, 2014 at 22:56
• Most standard models assume independence. A typical linear model is unbiased, even if the residuals are not independent (or are heteroscedastic), though. The problem would be with the tests / p-values of those models. For that there are heteroscedasticity (& potentially autocorrelation) consistent 'sandwich' errors. But I don't know enough yet about your thinking to know if that would be (part of) the answer to your question. Commented Dec 15, 2014 at 23:00
• Can you say more about what would constitute a "non-parametric" model for you & why you want one? Commented Dec 15, 2014 at 23:02

Using polynomials to fit curves is a standard feature of stats 102. However, for anything beyond the simplest and best behaved curves, polynomials are a poor choice (as you note). A better strategy is to use cubic splines. Your predictor space is divided into regions with a function fit to the entire space and additional functions fit to the region greater than the border (or 'knot'). Consider a case with only one $X$ variable / dimension fit with a linear spline. Let's say the range of $X$ is partitioned at $X = .7$, then:
$$X_{\rm spline} = \begin{cases} 0\quad &\text{if } X\le{.7} \\ X-.7\quad &\text{if } X>.7 \end{cases}$$ Then a multiple regression model is fit using the two variables $X$ and $X_{\rm spline}$. This is quite rudimentary, of course. It would be more typical to have, say, 5 knots instead of one, and you can fit polynomials (typically cubics) for each variable. This is a very powerful and flexible strategy for function approximation. Moreover, all the benefits of linear models come naturally with this approach. I have more information about this here: What are the advantages / disadvantages of using splines, smoothed splines, and Gaussian process emulators?
• How does this technique work for functions $f$ which are non-separable, eg. $f(x,y,z)=xyz$? I can see how to divide the predictor space into partitions, but would this mean that that the result wouldn't be guaranteed to be continuous? Commented Dec 19, 2014 at 14:54
• Just to be clear, let's suppose we have the (unknown) function $f(x,y,z)=xyz$. Are you proposing the following: Partition the 3d variable space into a number of cubes Fit a regression model to each cube. Presumably this is of the form $F=X+Y+Z$? Or what other predictor would I use Impose constraints on the regression so that the resulting curve was continuous Commented Dec 23, 2014 at 12:04