2
$\begingroup$

I was exploring the KL Divergence and came across some research about calculating it from samples. On stack-exchange, I found out that minimising the KL Divergence is equivalent to minimising the Sum of squares in Linear Regression.

To explore this further, I tried to implement a OLS and Ridge regression model for the Boston Housing Data-set and calculated the KL Divergence as mentioned in the papers. This gave me inexplicable results.

Firstly, the KL Divergence was negative for the estimator from Kullback-Leibler divergence estimation of continuous distributions. And instead of being negative for the OLS fit from the actual responses, it was negative for the Ridge fit from the ground truth, which is just contrary to the theoretical case.(I interpret negative KL Divergence as the fitted values being so good that the estimator just gave negative values, and would have given zero if I had the actual Distribution instead of samples from it).

Is there a clean explanation why this is happening?

PS: Just to explore the multidimensional capabilities of the estimators, I also passed in the entire data-matrix with the response/fitted value appended instead of just the responses to the KL Divergence estimators. This made the remaining 4 implementation of the estimator from Based on my understanding, this assumes that there is an underlying probability distribution which generated all the values for the covariates and the responses, and we calculate the KL Divergence between these distributions. I thought that this should not change anything because if we go back to the equivalence between OLS and KL Divergence, I imagine this as having a bunch of zeroes tacked onto the sum does not make a difference. Nevertheless, this experiment led to a negative value for all the four implementation of the estimator in the GitHub link.

Thus, coming back to the question posed earlier, how do I interpret the negative estimates of KL divergence, and why is it happening in this case.

References:

Estimators coded in python

Pérez Estimator

Paper for estimators coded in python

Results of calculation for uni-dimensional case

Estimator d_truth_ols d_truth_ridge d_ols_ridge
skl_efficient 0.05405258685484415 0.08663935806497296 0.04198067991782772
scipy_estimator 0.802908876382145 0.7347464259492821 0.6900877478021367
naive_estimator 0.05405258685484412 0.08663935806497293 0.04198067991782768
skl_estimator 0.05405258685484412 0.08663935806497293 0.04198067991782768
perez -0.06423632396883772 0.021712462670751664 0.05954526258402315

Results of calculations for multidimensional case (when I pass the entire data-matrix)-

Estimator Name KL(y_Ridge, y_Truth) KL(y_OLS, y_Truth) KL(y_Ridge, y_OLS) KL(y_OLS, y_Ridge)
skl_estimator -0.12739426 -0.22722966 -0.24495304 -0.41479071
naive_estimator -0.12739426 -0.22722966 -0.24495304 -0.41479071
skl_efficient -0.12739426 -0.22722966 -0.24495304 -0.41479071
scipy_estimator 7.73544109 6.74554664 7.61788232 6.55798559

Lastly, I just guess that the SciPy estimator is out of whack. Thanks a lot!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You interpreted negative KL Divergence as the fitted values being good to the point where the estimator gave you negative values. If I understood correctly, the estimator you used is unbiased, but known to have large variance. Approximating KLdiv(Q, P) by computing a Monte Carlo integral with integrands being negative whenever q(x) is larger than p(x) can naturally lead you to negative values of the estimator. Check for unbiased estimates with proven positivity, as this one from OpenAI's co-founder: Approximating KL Divergence

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.