This is such a difficult question to answer because 'data quality checks' is so broad, and even the definition of anomaly detection is not completely clear. Moreover, to give the best suggestions we would need to know a lot more about your use case. However will all that being said, I have prepared an answer which I hope raises interesting points which you could think about.
Brief description of Anomaly Detection
It can be difficult to summarise anomaly detection (note that outlier is often used as a synonym for anomaly).
The following summary is a reasonable one:
"Anomaly detection refers to the problem of finding patterns in data
that do not conform to expected behavior. These non-conforming
patterns are often referred to as anomalies, outliers, discordant
observations, exceptions, aberrations, surprises, peculiarities or
contaminants in different application domains. Of these, anomalies and
outliers are two terms used most commonly in the context of anomaly
detection; sometimes interchangeably."
(From Anomaly Detection: A Survey, Chandola, Banerjee, Kumar)
The following pithy description of an outlier is from Hawkins 1980 Identification of Outliers, may also be helpful to keep in mind.
"An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism"
(see Hawkins D., Identification of Outliers, Chapman and Hall, 1980.)
I'll give some 'textbook' applications of anomaly detection:
Examples of Anomaly detection:
Credit card fraud detection: Looking for anomalies(outliers) in credit card transaction data, for example unusual patterns in spending, such as an abnormally large purchase or spending in a new location. An anomaly detection model might detect this transaction and some action may be taken - for example the bank temporarily blocks the payment or contacts the customers.
Fault detection, for instance in manufacturing: We might monitor parameters like temperature, pressure, vibration and product dimensions in a production process. We can establish a baseline from historical data, and significant deviations from expected ranges or patterns can be flagged as anomalies or faults. e.g. sudden temperature spikes or irregular product dimensions. Anomaly detection methods can help us identify these anomalies (so we don't sell defective products), and allow us to take timely actions.
Brief description of 'Data Quality' checks
Defining data quality is difficult (even Wikipedia agrees!) - it depends a lot on what you are trying to do with the data as to how you would judge the quality. The same data may be high quality for one use case but low for another.
Broadly speaking, we might consider data quality to refer to the overally reliability, accuracy, completeness, consistency and relevance of data, ensuring that it is fit for its intended purpose and can be trusted for decision-making and analysis.
Examples of things to consider:
- How will the data be used? Age categories Child/Adult may be fit for some purposes, but for others we may need more granular ages.
- Missing data - are we missing data, how is the missing data recorded - are rows completely omitted or do they appear with some null value?
- Correctness. Is the data correct? Note that the data can be correct, and still be low quality.
Your specific problem and the overlap of the two
The relevance of anomaly detection techniques to your problem depends exactly on what data quality issues you are trying to find? Without more details it is impossible to precisely answer your question, however I will raise a few points you might like to think about.
You mention tables, it is possible to set up data quality tests for your tables (say in SQL) - which allow you to check for things such as null values or non-uniqueness of ids. (see dbt tests).
You mention statistical models/machine learning, and so I wonder whether what you are really driving at here is detecting anomalies in your data and assuming that these anomalies are symptoms of data quality issues. However I would caution you here to make sure you are familiar with the underlying processes in play here. For example in the Fraud detection example above, if you had a column in your table with user transacation amount, then your model might detect an unusually large value - but this value may well be correct, and so there is no data correctness issue here - the data may also have been delivered to you in near real time, so in many regards this is high quality data. On the other hand, if you were analysing data collected by human researchers of children's heights, and you noticed an usually large height of 113 metres, then you would probably conclude this is a genuine data quality issue (whereby the researcher has recorded in metres rather than centremetres).
- Can you use simple rules to highlight certain anomalous values. For example values which are physically impossible (e.g. the weight in kilograms of a component cannot be negative).
- Do you have some baseline data and a general understanding of what your data should look like?
-- You could use something as simple as a z-score to highlight values which are extreme in the sense that they are "far" from the mean value. In some contexts this might make sense but in others will be meaningless.
- How often is data recorded and when will you be running your checks (near real time? In batch at the end of the month?)
- You could compare two sets of data to see if there is a statistically significant difference in some statistic. However even if you find a difference, it does not necessarily mean there is a data quality issue.
- If you have time series data you could consider something like comparing the 'distance' of the current value with a rolling mean/median (see Hampel filters).
You should also think about what it would mean if you run some rule or model and find there is a potential data quality issue with a piece of data. Will a human review it? Will the data be omitted or will you replace it with some more 'suitable' value? (mean? median?) What is the cost of getting this wrong - for instance if you fail to detect incorrect data what would the consequence of this be on the business decisions? On the other hand, if you wrongly flag data as having some issue, what would the cost of that be (human operational costs?)
To summarise, my advice for you would be to try to understand the underlying process which generates your data and bear this in mind when considering what 'unusual' means and is my unusual data actually 'incorrect'.
