I am pulling in a handful of different datasets daily, performing a few simple data quality checks, and then shooting off emails if a dataset fails the checks.

My checks are as plain as checking for duplicates in the dataset, as well as checking if the number of rows and columns in a dataset haven't changed -- See below.

assert df.shape == (1016545, 8)
assert len(df) - len(df.drop_duplicates()) == 0

Since these datasets are updated daily and may change the number of rows, is there a better way to check instead of hardcoding the specific number?

Right now, for tables that change daily, I'm doing the following rudimentary check:

assert df.shape[0] <= 1016545 + 100
assert df.shape[0] >= 1016545 - 100

But obviously this is not sustainable.

For instance, one dataset might have only 400 rows, and another might have 2 million. Could I say to check within 'one standard deviation' of the number of rows from yesterday? How many previous days would I need to collect? And what would be the best way to perform this calculation?

Given the math, I should be able to implement it into code. Any suggestions are much appreciated. Us programmers are indebted to you math folks.


So currently there are 1016545 rows in that particular dataset. +-100 is just creating a range, since this particular table can change daily, it would be ok if the table had anything between and including 1016445 or 1016645. I am more concerned with sending out email alerts if we see a drastic change --> table rows drop to half or double.

Say this dataset is tracking the population of a city, where each row is the info of a person who lives in the city. The number of rows would change over time as people move to or leave the city. We may see a steady increase over a month, which would be fine. My alerts want to catch something drastic, thus I was thinking raising an alert if this number falls outside of one standard deviation would be a good place to start.

  • $\begingroup$ What is the second code chunk for? What does 100 stand for there? :) $\endgroup$
    – inmybrain
    Commented Feb 12, 2020 at 1:02
  • $\begingroup$ Added my edit. Let me know if you have any other questions. $\endgroup$
    – sanjayr
    Commented Feb 12, 2020 at 15:58
  • $\begingroup$ Thanks for the update, so you'd like to find a (drastic) change point of the number of rows. The 1-sd approach seems good as well, but you could make it specific to your problem by mingling with your previous data. $\endgroup$
    – inmybrain
    Commented Feb 13, 2020 at 1:02


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.