I am pulling in a handful of different datasets daily, performing a few simple data quality checks, and then shooting off emails if a dataset fails the checks.
My checks are as plain as checking for duplicates in the dataset, as well as checking if the number of rows and columns in a dataset haven't changed -- See below.
assert df.shape == (1016545, 8) assert len(df) - len(df.drop_duplicates()) == 0
Since these datasets are updated daily and may change the number of rows, is there a better way to check instead of hardcoding the specific number?
Right now, for tables that change daily, I'm doing the following rudimentary check:
assert df.shape <= 1016545 + 100 assert df.shape >= 1016545 - 100
But obviously this is not sustainable.
For instance, one dataset might have only 400 rows, and another might have 2 million. Could I say to check within 'one standard deviation' of the number of rows from yesterday? How many previous days would I need to collect? And what would be the best way to perform this calculation?
Given the math, I should be able to implement it into code. Any suggestions are much appreciated. Us programmers are indebted to you math folks.
So currently there are 1016545 rows in that particular dataset. +-100 is just creating a range, since this particular table can change daily, it would be ok if the table had anything between and including 1016445 or 1016645. I am more concerned with sending out email alerts if we see a drastic change --> table rows drop to half or double.
Say this dataset is tracking the population of a city, where each row is the info of a person who lives in the city. The number of rows would change over time as people move to or leave the city. We may see a steady increase over a month, which would be fine. My alerts want to catch something drastic, thus I was thinking raising an alert if this number falls outside of one standard deviation would be a good place to start.