kullback leibler divergence of empirical density and fitted density in R I have a vector and I used Pareto and Log-normal distribution to get a predicted density function. I am wondering is there any easy ways to calculate the KLD between the true distribution and fitted distribution?
I guess I may need a discrete form, though the fitted density is continuous?
Below is the graph of my data and fitted distributions.

 A: To calculate the KLD estimate between the log-normal distribution and the Pareto distribution with respect to the empirical distribution, you must turn them into discrete values. The definition of the KLD in discrete form is
$$D_{KL}(P||Q) = \sum_i p(i) \log \frac{p(i)}{q(i)} $$
where P represents the true distribution and Q represents the theoretical distribution. In an essence, it is about the average information gained based on a prior Q. 
To do such a task in R, I used the following code: 
library(fitdistrplus); library(actuar); library(VGAM)
set.seed(123456)
a     <- rexp(1000, 0.25)
coefs <-coef(fitdist(a, 'lnorm'))
lnorm <- dlnorm(seq(min(a),max(a), by = .001), meanlog = coefs[1], sdlog = coefs[2])
coefp <- coef(fitdist(a, 'pareto', start = list(shape = 1, scale = 1)))
paretox<- dpareto(seq(min(a),max(a), by = .001), coefp[1], coefp[2])
h  <- hist(a, breaks = 30, freq = F, ylim = c(0,.35), xlim = c(0, 30))
lines(seq(min(a),max(a), by = .001), lnorm, col = 'blue', lwd = 3)
lines(seq(min(a),max(a), by = .001), paretox,col = 'darkgreen',lwd = 3)

pln <- dlnorm(h$mids, coefs[1], coefs[2])
ppar<- dpareto(h$mids, coefp[1], coefp[2])
trus<- h$density
pln <- pln[which(trus>0)]; ppar <- ppar[which(trus>0)]
trus<- trus[which(trus>0)]
KLD_lnorm <- sum(trus*log10(trus/pln))
KLD_pareto<- sum(trus*log10(trus/ppar))

> KLD_pareto
[1] 0.005314876
> KLD_lnorm
[1] 0.0150227

The code above first creates an exponential random variable a and the finds the lnorm or log-normal coefficients used in the dlnorm function. Then the program finds the coefficients of the pareto function, scale and shape as used the dpareto function. Then we plot the histogram and corresponding distributions based on a log-normal and a Pareto distribution. The reason we store the hist function is because within this function, there are valuable pieces of information, like $mids and $density. This allows us to extract the midpoints of the bins and the density at each midpoint. Then we find the values of the log-normal density and the Pareto density at the midpoints. There may be a zero-value issue of some of the density values, so we need to find them in the trus data and remove them because you cannot take the log(0). Then you plug your values into the KLD formula as above. As you can see, the KLD for the Pareto distribution in my example performed better than the log-normal. 
