I work as an analyst at a company that is in the midst of transitioning to more cloud-friendly tools. That means we've gone from using SAS/TD SQL (from my actual analyst days) to working with Python and Snowflake, etc. Unfortunately, that also means some old proprietary tools have been retired, including ones I wish we still had. Happily, though, we're into the world of open-source software, so hoping that makes up for it :)
The tool I'm looking to replicate was used to compare two datasets with the same variables on customers. More precisely, we often used it to identify what key characteristics changed between, say, two consecutive (calendar) quarters of customers selecting the same product. Its features included:
Automatically bucketing categorical/discrete (don't remember if it differentiated the two) and continuous variables together
Rank ordering variables within each bucket by relevant statistic (Chi-Square or KS) from most to least "different" according to those tests
Producing relevant plots for each variable (e.g., bar chart for categorical variables showing the distros between samples, cumulative distribution function for continuous variables to give context to the KS statistic)
Min/max/mean/median statistics for each variable to go with the plots
Its output was an HTML report, containing a homepage with the bucketed and ranked variables from the samples in their own tables with the key statistics (KS, missing, total count, etc.). Clicking on an individual variable would bring up a new HTML page with the distribution chart + some additional stats on that variable (mean/median/mode/range/max/min in each sample). I believe input allowed for downsampling as well as stratification on at least one dimension, if normalization against a variable in the reference sample was desired.
All of the above seems pretty doable to replicate and was immensely useful to me (or felt like it). I have a few questions for those here:
1.) Is anyone aware of packages like this that exist today in python (or R)? This tool was effectively a nicer proc compare; function in SAS.
2.) If not, I'd love suggestions on the essential packages/libraries needed to build this myself. I've been looking to dust off my Python anyway, and this would be a good way to do it. I've dug up the function for computing KS in SciPy, I'm assuming I'd use MatPlotLib for CDF and bar charts. Would maintaining an HTML output be advisable? What package(s) would be useful to do something like that? Advice on the plethora of little nuisances to come with this project e.g., handling a couple missing fields in one sample?
3.) More generally, is this a sane thing to want? Are there better ways to approach this problem? I understand that there are other methods available (residual analysis to understand where our models are blind to the shifts in performance; clustering to identify multivariate nodes where populations are changing over time; matching algorithms to try to identify like and non-like populations; etc.) to answer these "what's different?" type questions more elegantly, but this tool proved super lightweight & efficient in the past to give a good starting point. Would I be better off using one of those other methods and/or just building a regression model and rank ordering variable importance with the target being from which sample account came? I figure there's more burden required to deal with collinearity in that case when I'm seeking something easy at which to throw the kitchen sink to start.
Suggestions along any of the above lines would be wonderful!