4
$\begingroup$

What statistical methods out there that will estimate the probability density of data as it arrives temporally?

This is the situation I have:

I need to estimate the pdf of a multivariate dataset; however, new data arrives over time and as the data arrives the density estimation must update.

What I have been using so far is kernel estimations by storing all the data and computing a new kernel density estimation with every update of new data; however, I cannot keep up with the amount of data needed to be stored. Therefore, I need a method that will keep track of the overall pdf/density estimation then be able to update itself when new data arrives. This sort of sounds like it may need some sort of Bayesian method but I am not sure

Any suggestions would be really helpful.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You could store your data as counts in intervals, where the intervals are substantially narrower than the bandwith you are using for smoothing your kernel. Then you just need to remember as many numbers as there are intervals. $\endgroup$
    – Henry
    Commented Jan 12, 2015 at 8:21
  • $\begingroup$ I have posted a somewhat related question (with bounty!) here: mathoverflow.net/questions/382260/… $\endgroup$
    – Matthieu Latapy
    Commented Feb 1, 2021 at 17:39

2 Answers 2

Reset to default
6
$\begingroup$

I have become curious about the topic and performed a brief research. It appears that, generally, the taxonomy of approaches to kernel density estimation (KDE) for data streams consists of two large groups: offline and online approaches. The offline approaches include so called CF-kernel and KD-kernel methods, while the online approaches include grid-based, sample-based and cluster-based groups of methods. More details about this taxonomy as well as one of the methods (called adaptive KDE), can be found in this research paper.

M-Kernel method is the core of the cluster-based approach to KDE for streaming data and is discussed in various research papers, for example in this and this. Finally, an interesting and novel method, implementing KDE for data streams, based on self-organizing maps (SOM), has been proposed recently. It is called SOMKE and is described in this research paper. Hope this helps.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you very much. I had come across M-kernel before and was considering it for a long time. $\endgroup$
    – user2909415
    Commented Jan 12, 2015 at 9:05
  • $\begingroup$ @user2909415: You're very welcome. I'm glad that you liked my answer. Feel free to upvote. Now you have a wider range of options, not only M-kernel :-). $\endgroup$
    – Aleksandr Blekh
    Commented Jan 12, 2015 at 9:30
0
$\begingroup$

If you're willing to store some samples of the data, you can also try reservoir sampling. Basically, you can build one or more reservoirs of samples and run the usual kernel density estimation algorithms on top your samples. The challenge then becomes how to figure out the number of reservoirs and sample sizes, and what is the confidence interval on your estimates.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.