Smoothing when standard errors are known/estimated

Question

I have a set of estimates of function values, along with estimates of their standard errors. To somewhat simplify matters: for $x$ running from $1$ up to $70$ in (integer) steps (in fact a parameter in model selection), I have an estimate of the matching $y$-value (in fact an AUC) and an estimate of its SE. I cannot make assumptions on the form of the curve (it's not even certain that it's unimodal, for example), other than that it should be smooth on the scale of my $x$ values.

I am looking for a way to translate these to a more smooth form, because the method I am using tends to create the occasional spike that I hope to circumvent.

Now, most smoothing algorithms only use the $y$ values, which, in this case, seems to be a waste of information (i.e. not using the SE). In addition, I would also like to smooth out an error band from the error flags.

Somewhat to my surprise, I have not found a technique that provides this. Have I not looked well enough? I would also like to perform this in R, so if anybody has a package that does the trick, I'd be very happy.

Answer 1

You can account for the differing standard errors using inverse-variance weights. You would weight each observation by the inverse of its variance, i.e. one over the square of its standard error. The simplest case is weighted least squares but similar principles apply to most other types of regression models. There are many possible smoothing methods such as smoothing splines or local polynomials.. the choice depends on what your data looks like, what you want to do with the results and what software you have available. Any method should be able to produce confidence or prediction bands around the fitted curve.

I'm not very familiar with what's available in R; I see the lm function allows a weights argument and the smooth.Pspline function of the pspline package has a w argument. I assume other functions and packages have something similar - perhaps someone else can expand (I'm more familiar with Stata which allows weights in virtually all its regression models).