Estimating the functional form of the slowly time-varying variance of a Gaussian process Consider the following simple set-up. In the interval $[0, 1]$ we are observing the realizations of independent normally distributed random variables at times $t_1,\ldots, t_N$. The r.v. $X(t)$ has mean 0 and standard deviation $\sigma(t)$, where all is known about $\sigma(\cdot)$ is that it is smooth, say at least differentiable. The goal is to estimate $\sigma$ so that the expected loss function (left to the reader to choose) is minimized.
I can think of many practical approaches, e.g. using kernels or approximating in a given basis, or penalizing the MLE; but is there any published literature on this simple problem? If not, what formal approach would you take?
 A: In functional data analysis, people often use penalties of the form 
$$ \int_{D} [f^{(m)}(x)]^{2} dx $$ 
to ensure that an estimate of $f$ is smooth. Here $D$ is the domain of the function and $f^{(m)}(x)$ is the $m$'th derivative of $f$. In my own research I've found $m = 2$ to be useful. In your case the log-likelihood is 
$$ -\frac{1}{2} \left( 2 \sum_{i=1}^{N} \log \big( \sigma(t_{i}) \big) + \sum_{i=1}^{N} \frac{x_{i}^{2}}{\sigma^{2}(t_{i})} \right) $$ 
An estimator that minimizes
$$ 
\left( 2 \sum_{i=1}^{N} \log \big( \sigma(t_{i}) \big) + \sum_{i=1}^{N} \frac{x_{i}^{2}}{\sigma^{2}(t_{i})} \right) + \lambda \int_{0}^{\infty} [\sigma^{(2)}(t)]^{2} dt $$ 
(or, equivalently, maximizes the penalized log likelihood) for an appropriately chosen tuning parameter, $\lambda$, will be smooth. This is a differentiable (on the interior of the parameter space) objective function of $\sigma(t_{1}), ..., \sigma(t_{K})$ (where $K$ is the number of unique time points) so it can be optimized using some standard approaches. Although, the potential non-convexity of the objective function could make starting values crucial for any optimization algorithm. 
I've found that choosing $\lambda$ based on some model selection criteria such as AIC performs pretty well although there are many other potential choices. If you're going to go that route, the effective number of parameters in the model (which is required to calculate AIC) can be tricky to determine. As $\lambda \rightarrow 0$, you have the non-parametric MLE, so the number of parameters is equal to the number of unique times observed. As $\lambda \rightarrow \infty$, you have a perfectly linear fit (since the penalty will be 0 only when $\sigma(t)$ is linear and approaches $\infty$ otherwise) so there effective number of parameters is 2. Between these two extremes the effective number of parameters is not so obvious. Shedden and Zucker (2008)
http://www.ncbi.nlm.nih.gov/pubmed/19956348
use the AIC approach to choosing smoothness and derive an approximation to the effective number of parameters under penalization - that may be useful to look at. 
