In the example figure below, KL-divergence is being used to measure how far the distribution of different parameterizations of Poisson are from an empirical distribution (real data). The minimum of the curve, where KLD is closest to 0, is where we identify the optimal parametric model to be used as a fit on the real data.

enter image description here

If extending this single-variable process to multiple variables, each with their own empirical distribution that requires a different optimal parameter value for the parametric (fitted) distribution, does there exist a rate of change performance measure for KL-divergence? so that individual rates can be compared across multiple variables

The idea is to identify which of the variables descends fastest to its respective minimum for the same range of parameterizations applied to the other variables. For this I'm thinking the first derivative of KL-divergence would reflect its rate of change, or gradient?

a related discussion was for multinomial data and focused alot on Bayesian concepts and moments, so I'm looking for a more direct rate of change approach without moments, for real continuous data (Poisson was only a visual example), given the simple aim of incremental re-parameterization evaluation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.