1
$\begingroup$

I am pulling in a handful of different datasets daily, performing a few simple data quality checks, and then shooting off emails if a dataset fails the checks.

My checks are as plain as checking for duplicates in the dataset, as well as checking if the number of rows and columns in a dataset haven't changed -- See below.

assert df.shape == (1016545, 8)
assert len(df) - len(df.drop_duplicates()) == 0

Since these datasets are updated daily and may change the number of rows, is there a better way to check instead of hardcoding the specific number?

Right now, for tables that change daily, I'm doing the following rudimentary check:

assert df.shape[0] <= 1016545 + 100
assert df.shape[0] >= 1016545 - 100

But obviously this is not sustainable.

For instance, one dataset might have only 400 rows, and another might have 2 million. Could I say to check within 'one standard deviation' of the number of rows from yesterday? How many previous days would I need to collect? And what would be the best way to perform this calculation?

Given the math, I should be able to implement it into code. Any suggestions are much appreciated. Us programmers are indebted to you math folks.

EDIT:

So currently there are 1016545 rows in that particular dataset. +-100 is just creating a range, since this particular table can change daily, it would be ok if the table had anything between and including 1016445 or 1016645. I am more concerned with sending out email alerts if we see a drastic change --> table rows drop to half or double.

Say this dataset is tracking the population of a city, where each row is the info of a person who lives in the city. The number of rows would change over time as people move to or leave the city. We may see a steady increase over a month, which would be fine. My alerts want to catch something drastic, thus I was thinking raising an alert if this number falls outside of one standard deviation would be a good place to start.

$\endgroup$
  • $\begingroup$ What is the second code chunk for? What does 100 stand for there? :) $\endgroup$ – inmybrain Feb 12 at 1:02
  • $\begingroup$ Added my edit. Let me know if you have any other questions. $\endgroup$ – sanjayr Feb 12 at 15:58
  • $\begingroup$ Thanks for the update, so you'd like to find a (drastic) change point of the number of rows. The 1-sd approach seems good as well, but you could make it specific to your problem by mingling with your previous data. $\endgroup$ – inmybrain Feb 13 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.