Bayesian updating of nonparametric estimate of distribution Is there any way to perform "bayesian updating" of a nonparametric estimate of some distribution (say, a kernel density estimation) in light of a new set of observed values?
 A: *

*Kernel density estimate is
$$
f(x) = \frac{1}{n} \sum_{i=1}^n \, K_h(x - x_i)
$$
so to update it you don't need to do anything special, if a new $x_{n+1}$ point comes, just increment $n$ by one and append the array of $x_i$'s by a new value.


*We usually don't fit kernel density by memorizing and storing all the data, but we use tricks to approximate the kernel density to save both space and computation time. Kernel density with Gaussian kernels is in fact a Gaussian mixture with equal weights. You could collapse the components that are close to each other, to make it a standard mixture with not necessarily equal weights and variances. There is no closed-form solution for Bayesian updating of a mixture distribution, we usually do this by either optimization or approximating the distribution, e.g. using MCMC sampling. The problem with estimating parameters of mixture models is label switching so the usual approaches to optimization and sampling do not work as expected. In the slides for the Learning Algorithms for Gaussian Mixture Models talk Kazuyoshi Yoshii discusses different approaches for estimating the parameters of Gaussian mixtures that include optimization using E-M algorithm, sampling, and variational inference (expectation-expectation). Those algorithms would, unfortunately, need all the data and do not enable you for a simple update of pre-estimated parameters.


*It is not a Bayesian approach, but you could use online $k$-means algorithm as inspiration to update the estimate given a new data point.

$$\begin{align} j &:= \operatorname{arg\,max}_j \; K_h(x - x_j) \\ n_j
 &:= n_j + 1 \\ x_j &:= x_j + \tfrac{1}{n_j} ( x_i - x_j) \\
 \end{align}$$
then your density estimate is
$$ f(x) = \sum_{j=1}^k \, \frac{n_j}{N} \, K_h(x - x_j) $$



*If instead of kernel density you are using a histogram to estimate the distribution, or an empirical distribution for discrete data, just treat it as the conjugate Dirichlet-categorical model to update the probabilities.
