How do I judge the accuracy of an estimated density function?

Question

I want to use a few different methods to create different probability density functions. These pdfs change over time as different data is used (rolling historical window used as training data). I want to find out which method has produced PDFs that most consistently represented the final outcomes. Is there a general method or formula used for this?

My initial thought is that I can make something similar to a Q-Q Plot -- comparing the estimated values with the actual values that happened, and see if they come from the same distribution.

Answer 1

The simplest quantitative way is to evaluate them on held-out data. For example, with the PDF chosen based on data points up to $n$, evaluate its density on $n+1$; then update the model based on $n+1$ and evaluate on $n+2$, and so on. Add up the log densities for a total score.

Plots are also good for understanding: plot the predictive distribution and the point that actually occurred for a bunch of samples, some random and some the worst predictions, to try to understand the model and its errors better.