When to use distance between distributions instead of using median based statistics? I have a problem where I have to compare the effect of X ( univariate random variable) on distribution of Y (univariate random variable) between 2 different cases. Y is not following a Normal distribution. I am not sure whether to Mean/ median-based statistics (such as ANOVA) or use a distance between distribution metric (such as KL divergence). 
 A: I have never seen the KL divergence used as a test statistic. I think this is because it is not sensitive to the sample size, i.e. if the two distributions are different, then a larger sample size doesn't give a larger statistic than a smaller one. Most statistical tests for the difference between two groups concern their means only. (It is also possible to test for difference in medians, just less common.) The reason this is the case, I believe is to do with interpretability. If two samples differ in their distributions, then one is then bound to ask --- in what way? Is the mean different, or the variance, or the skew, etc? Another reason is mathematical niceness. Tests of means are the most straightforward and well known, especially when assuming underlying Normality. It is also easily interpretable. Furthermore, a difference in means implies a difference in distributions anyway. 
It is also common to use the non-parametric tests such as Mann-Whitney U test or Kruskal-Wallis to test for differences in groups. These tests actually test for difference in distributions. However, they are not very sensitive to departures in other moments such as variance and skew. 
A: Consider the following two samples of sizes 20 and 25,
respectively. Descriptive statistics and boxplots follow:
summary(x1); sd(x1); length(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.2211  0.3262  0.4296  0.4446  0.5710  0.7330 
[1] 0.1534525  # SD x1
[1] 20         # n for x1
summary(x2); sd(x2); length(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.6565  0.8064  0.9148  0.9348  1.0124  1.2987 
[1] 0.1723941
[1] 25
boxplot(x1,x2, col="skyblue2", notch=T)


Boxplots show slight upward skewness, especially for x2. Normal probability plots seem nonlinear.
So the data may not be from normal populations.
The median of the first sample is below the median of the second sample.
The notches in the sides of the boxes in the boxplots above suggest
that the difference in medians is significant. 
Boxplots show slight upward skewness. Normal probability plots (especially for x2) seem nonlinear.
So the data may not be from normal populations. Samples are a bit
too small for reliable results from a Shapiro-Wilk test of normality, but
neither sample is detected as nonnormal by the test.
par(mfrow=c(1,2))
 qqnorm(x1); qqline(x1, col="blue")
 qqnorm(x2); qqline(x2, col="blue")
par(mfrow=c(1,1))


shapiro.test(x1)$p.val
[1] 0.2048991
shapiro.test(x2)$p.val
[1] 0.1861775

The departure from normality is probably within the scope of the
robustness of a t test, but we use a nonparametric test instead.
A two-sample
Wilcoxon (rank sum) test finds a highly significant difference in location.
 wilcox.test(x1, x2)

    Wilcoxon rank sum test

data:  x1 and x2
W = 1, p-value = 1.262e-12
alternative hypothesis: 
   true location shift is not equal to 0

Notes: (1) In practice, you can only say what you find in the data.
It seems clear from the data that these two samples did not come
from the same population.  However, in this case the data were
simulated as shown below, so we know for sure that the two samples
are nonnormal and that they differ. Sample x2 is from a population shifted to the right (by $0.5$) of
the population from which x1 was sampled.
set.seed(602)
x1 = round(rbeta(20, 5, 6), 4)
x2 = round(rbeta(25, 5, 6) + .5, 4)

(2) A Welsh two-sample t test would also have found a significant
difference (in population means). There are no outliers and skewness
is quite slight, so it is not surprising that the t test comparing the two samples gives a
useful result.
t.test(x1,x2)$p.val
[1] 7.842386e-13

