I am pulling in a handful of different datasets daily, performing a few simple data quality checks, and then shooting off emails if a dataset fails the checks.
My checks are as plain as checking for duplicates in the dataset, as well as checking if the number of rows and columns in a dataset haven't changed -- See below.
assert df.shape == (1016545, 8)
assert len(df) - len(df.drop_duplicates()) == 0
Since these datasets are updated daily and may change the number of rows, is there a better way to check instead of hardcoding the specific number?
Right now, for tables that change daily, I'm doing the following rudimentary check:
assert df.shape[0] <= 1016545 + 100
assert df.shape[0] >= 1016545 - 100
But obviously this is not sustainable.
For instance, one dataset might have only 400 rows, and another might have 2 million. Could I say to check within 'one standard deviation' of the number of rows from yesterday? How many previous days would I need to collect? And what would be the best way to perform this calculation?
Given the math, I should be able to implement it into code. Any suggestions are much appreciated. Us programmers are indebted to you math folks.
EDIT:
So currently there are 1016545 rows in that particular dataset. +-100 is just creating a range, since this particular table can change daily, it would be ok if the table had anything between and including 1016445 or 1016645. I am more concerned with sending out email alerts if we see a drastic change --> table rows drop to half or double.
Say this dataset is tracking the population of a city, where each row is the info of a person who lives in the city. The number of rows would change over time as people move to or leave the city. We may see a steady increase over a month, which would be fine. My alerts want to catch something drastic, thus I was thinking raising an alert if this number falls outside of one standard deviation would be a good place to start.
