I am a product manager for a data analytics startup, and on our platform our clients end up with large tables of data about their customers which has many attributes, which are of mixed datatype: some attributes are numeric, some date, some boolean, and some categorical with arbitrary cardinality. These users will create various segments (samples) of that population, and we want to be able to answer the question: Which of these attributes are the most different between this segment and the full population? This will help them identify interesting places to explore further, to identify relationships or characteristics of the subgroup.
In this case "most different" can have a meaning that is somewhat up to us, but should be quantifiable and meet, as best we can, a few desirable characteristics:
- Must yield a comparable difference magnitude for each attribute of a given datatype. We don't need these results to be comparable across different subgroups, just comparable across attributes of the given subgroup (when compared with the population).
- Be comparable across datatypes, or have a variation that yields comparable results for each one. If this comparison is a little noisy between, say, numerics and categoricals, it's still OK as long as it's roughly on the same scale. But it's important that we be able to sort of compare apples and oranges here.
- Function reasonably well regardless of how small a fraction of the full population is in the subgroup being compared. Though "small" segments will still have an N of decent size (i.e. thousands), we're more talking about this being about equally valid for a 10% sample and for an 80% sample of a million-row population, and still pretty good for a 1% sample.
So for example, for numeric attributes, we could take median or mean and just compare those and say "the segment is 30% above the population average for Attribute A, 25% below for Attribute B", etc. But there is no equivalent to that measure for categorical attributes (and it's a bit sketchy with dates, too - you can't very well say the median date for the segment is "15% above or below" that of a population). So we need some measure of "difference" that works for all datatypes, or multiple measures each of which work well for different datatypes but whose output can be deemed roughly comparable.
A few approaches have suggested themselves in our research, but I'm probably missing a few, and I'm somewhat unable to evaluate the tradeoffs between them, so I'm wondering what an actual statistician would recommend here. Choices include:
- A Pearson's Chi-Squared test, which will yield a p-value for categorical data which is nicely comparable across categorical variables. For boolean variables, there are lots of 2x2 variations on it available (e.g. would Yates' correction make the resulting p-values more comparable to higher-cardinality category variables, or less?). And making this comparable to any "difference metric" for numeric data seems harder.
- Similar things can be said for a G-test, which will yield a value G that is comparable to a chi-squared statistic and operates on categorical and boolean data. But it's more designed for a goodness-of-fit test than what we're doing which is a homogeneity test, and I don't know whether there's an equivalent choice for numeric data.
- On the other hand, the related concept of Kullbeck-Leibler divergence seems to be specifically designed to quantify the difference between two distributions, and may even be applicable to both discrete and continuous distributions (i.e. both categorical and numeric data). And I don't know enough to say whether KL divergence as a measure is fit-for-purpose here.
- For numeric data, a lot of metrics also yield a p-value, e.g. a regular-ol' T-test, a two-sample Kolmogorov–Smirnov test (against a cumulative distribution), the Mann-Whitney U-test, etc.
- The Mann-Whitney U-test yields a p-value, and helpfully also functions well with date values, since even though dates don't represent a quantity, you can at least compare one as being greater or lesser than another. Brunner-Munzel might be an improvement for our purposes, I'm not sure. But this assumes the values are comparable themselves (can say one is greater or lesser than another), which is not the case for boolean or categorical data.
- Then there's Fisher's Exact Test and its relatives, which seem principally designed for boolean data but seems extensible to any categorical attribute, and yields a p-value like the others, but it's unclear whether it's more comparable or less.
- Standardized Mean Difference or some other measure of effect size might be helpful in quantifying the difference between sample and population distribution, but seems poorly suited to categorical data.
One issue with any approach that yields a p-value is that I'm not sure how valid it is to say that if testing Attribute A yields a p-value of X and Attribute B yields a p-value of 2X, that the sample is "2x as distinct" from the population on A as it is on B. The underlying meaning of p-values is a probability that the sample could have been obtained by chance (or that the two groups were obtained randomly from the same theoretical distribution), so if you squint maybe you can present it as a quantification of degree of difference. But these statistics are using very different methods to arrive at a p-value, and if we're not using them for a threshold screen, the way they were designed for, I'm not sure if my off-label intended use is even remotely valid. Maybe it is for some of them.
Anyway I'd be grateful for any suggested approach here. Thanks!
