2
$\begingroup$

This question already has an answer here:

In trying to understand the Kullback-Leibler Divergence I conceive it as a metric that if minimized would make the Approximation PDF Q as close as possible to the True PDF P either in the neighborhoods of X where P places the highest probability mass (K(P||Q)) or in the neighborhoods of X where Q places the highest probability mass (K(Q||P)). So it is like saying to Q "try your best to approximate P where P thinks are the most probable neighborhoods" or inversely "try to approximate P the best in the neighborhoods you believe are the most probable."

On the other hand, trying to arrive to an intuitive understanding of the Kolmogorov-Smirnov statistic, I view it as a decision to underplay local divergences between two probability distributions functions in favor of the 'larger picture' which is reflected in the divergence of their respective Cumulative Distributions.

I wonder if these thoughts make sense, and how you would like to add to them with your own intuitive interpretations of the two Divergence 'metrics'.

In a more practical level when and how do you believe Kullback-Leibler Divergence and Kolmogorov-Smirnov statistic should be used?

A case in hand might be to compare the predicted probabilities by a classifier --say Logistic Regression-- to the empirical ones using the Validation set. Consequently compute the divergence between the two distributions --predicted and empirical-- using an appropriate divergence 'metric'. Finally use this information to select a model that best approximates the conditional probability P(Y|X). Which metric should we use and why?

$\endgroup$

marked as duplicate by Michael Chernick, kjetil b halvorsen, jbowman, AdamO, Robert Long Jun 12 '18 at 8:18

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.