# Bayesian updating of nonparametric estimate of distribution

Is there any way to perform "bayesian updating" of a nonparametric estimate of some distribution (say, a kernel density estimation) in light of a new set of observed values?

• For KDE, would you not just do the estimate with the $n+1$ points that you have now instead of the $n$ points you had before?
– Dave
Jun 1, 2021 at 13:13
• Possibly - but I'd like to see how that'd actually be justified with respect to Bayes theorem, and whether there's anything that can be done to systematically give higher weights to new data in an interpretable way. Jun 1, 2021 at 13:21
• Additionally, re-computing the entire estimate can be prohibitively costly. Jun 1, 2021 at 13:33
• Maybe if we estimate the distribution with $\exp \circ f$ where $f \sim \mathcal{GP}(0, k)$? Without an analytic posterior though the updating might not be easy
– jld
Jun 1, 2021 at 15:13
• Is this helpful stats.stackexchange.com/questions/473000/… ?
– Tim
Jun 5, 2021 at 17:05

• $$f(x) = \frac{1}{n} \sum_{i=1}^n \, K_h(x - x_i)$$

so to update it you don't need to do anything special, if a new $$x_{n+1}$$ point comes, just increment $$n$$ by one and append the array of $$x_i$$'s by a new value.

• We usually don't fit kernel density by memorizing and storing all the data, but we use tricks to approximate the kernel density to save both space and computation time. Kernel density with Gaussian kernels is in fact a Gaussian mixture with equal weights. You could collapse the components that are close to each other, to make it a standard mixture with not necessarily equal weights and variances. There is no closed-form solution for Bayesian updating of a mixture distribution, we usually do this by either optimization or approximating the distribution, e.g. using MCMC sampling. The problem with estimating parameters of mixture models is label switching so the usual approaches to optimization and sampling do not work as expected. In the slides for the Learning Algorithms for Gaussian Mixture Models talk Kazuyoshi Yoshii discusses different approaches for estimating the parameters of Gaussian mixtures that include optimization using E-M algorithm, sampling, and variational inference (expectation-expectation). Those algorithms would, unfortunately, need all the data and do not enable you for a simple update of pre-estimated parameters.

• It is not a Bayesian approach, but you could use online $$k$$-means algorithm as inspiration to update the estimate given a new data point.

\begin{align} j &:= \operatorname{arg\,max}_j \; K_h(x - x_j) \\ n_j &:= n_j + 1 \\ x_j &:= x_j + \tfrac{1}{n_j} ( x_i - x_j) \\ \end{align}

$$f(x) = \sum_{j=1}^k \, \frac{n_j}{N} \, K_h(x - x_j)$$