Bayesian updating of nonparametric estimate of distribution

Is there any way to perform "bayesian updating" of a nonparametric estimate of some distribution (say, a kernel density estimation) in light of a new set of observed values?

• For KDE, would you not just do the estimate with the $n+1$ points that you have now instead of the $n$ points you had before? – Dave Jun 1 at 13:13
• Possibly - but I'd like to see how that'd actually be justified with respect to Bayes theorem, and whether there's anything that can be done to systematically give higher weights to new data in an interpretable way. – user3716267 Jun 1 at 13:21
• Additionally, re-computing the entire estimate can be prohibitively costly. – user3716267 Jun 1 at 13:33
• Maybe if we estimate the distribution with $\exp \circ f$ where $f \sim \mathcal{GP}(0, k)$? Without an analytic posterior though the updating might not be easy – jld Jun 1 at 15:13
• Is this helpful stats.stackexchange.com/questions/473000/… ? – Tim Jun 5 at 17:05

• $$f(x) = \frac{1}{n} \sum_{i=1}^n \, K_h(x - x_i)$$

so to update it you don't need to do anything special, if a new $$x_{n+1}$$ point comes, just increment $$n$$ by one and append the array of $$x_i$$'s by a new value.

• We usually don't fit kernel density by memorizing and storing all the data, but we use tricks to approximate the kernel density to save both space and computation time. Kernel density with Gaussian kernels is in fact a Gaussian mixture with equal weights. You could collapse the components that are close to each other, to make it a standard mixture with not necessarily equal weights and variances. There is no closed-form solution for Bayesian updating of a mixture distribution, we usually do this by either optimization or approximating the distribution, e.g. using MCMC sampling. The problem with estimating parameters of mixture models is label switching so the usual approaches to optimization and sampling do not work as expected. In the slides for the Learning Algorithms for Gaussian Mixture Models talk Kazuyoshi Yoshii discusses different approaches for estimating the parameters of Gaussian mixtures that include optimization using E-M algorithm, sampling, and variational inference (expectation-expectation). Those algorithms would, unfortunately, need all the data and do not enable you for a simple update of pre-estimated parameters.

• It is not a Bayesian approach, but you could use online $$k$$-means algorithm as inspiration to update the estimate given a new data point.

\begin{align} j &:= \operatorname{arg\,max}_j \; K_h(x - x_j) \\ n_j &:= n_j + 1 \\ x_j &:= x_j + \tfrac{1}{n_j} ( x_i - x_j) \\ \end{align}

then your density estimate is

$$f(x) = \sum_{j=1}^k \, \frac{n_j}{N} \, K_h(x - x_j)$$

• If instead of kernel density you are using a histogram to estimate the distribution, or an empirical distribution for discrete data, just treat it as the conjugate Dirichlet-categorical model to update the probabilities.

• Great answer! Thanks for the links, in particular - I've got a fair bit of reading to do now. The online approach has the benefit of being extremely simple, but it presents us with the fairly difficult problem of determining how much past data to include in the estimate if we have reason to believe that our distribution is changing in time. Are there any good methodologies for this? – user3716267 Jun 6 at 14:25
• @user3716267 I didn't find any resources I could recommend. You can try yourself googling for "online kernel density estimation" and related phrases. If you find something, it'd be great if you could share (e.g. answering your own question). – Tim Jun 7 at 6:27