I am comparing customer ratings at origination and at default. My data looks like this
Customer ID Rating at Origination Rating at Default 1 7 6 2 4 13 3 6 12 ...
I have 483 unique customers, and a rating is an integer between 1 and 15. I would like to know whether the ratings at origination and those at default are from the same distribution.
1) The K-S Test
I am not sure whether the ties will invalidate the result of the K-S test. Nevertheless, the result of the K-S test is
ks.test(data$ORIG_RATING,data$DEF_RATING) Two-sample Kolmogorov-Smirnov test data: data$ORIG_RATING and data$DEF_RATING D = 0.072464, p-value = 0.1582 alternative hypothesis: two-sided
Warning message: p-value will be approximate in the presence of ties
According to the accepted answer of this question, K-S test can be used when the ties are not heavy. How can I tell whether my ties are heavy or not?
2) The Chi Square Test
chisq.test(data$ORIG_RATING, data$DEF_RATING) Pearson's Chi-squared test data: data$ORIG_RATING and data$DEF_RATING X-squared = 2287, df = 99, p-value < 2.2e-16
Warning message: Chi-squared approximation may be incorrect
3) Permutation Tests
According to the accepted answer of this question,
install.packages("afex") library(afex) compare.2.vectors(data$ORIG_RATING,data$DEF_RATING) $parametric test test.statistic test.value test.df p 1 t t -2.042615 964.0000 0.04136216 2 Welch t -2.042615 957.5903 0.04136398 $nonparametric test test.statistic test.value test.df p 1 stats::Wilcoxon W 108548.500000 NA 0.05839797 2 permutation Z -2.039266 NA 0.04277000 3 coin::Wilcoxon Z -1.892815 NA 0.05724000 4 median Z -1.029688 NA 0.33503000
These tests seem to reveal that my result is really at the border, but which one should I trust?
My Question: Can the fact that my observations are paired and bounded integers be used to make such a comparison more accurate? Are there any tests designed specifically for data like this?