What is a consistency check?

Question

I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is appreciated.

Answer 1

To chl's list, which focuses on frank data processing errors, I would add checks for subtler errors to address the following questions and issues (given in no particular order and certainly incomplete):

(1) Assuming database integrity, are the data reasonable? Do they roughly conform with expectations or conventional models, or would they surprise someone familiar with similar data?

(2) Are the data internally consistent? For example, if one field is supposed to be the sum of two others, is it?

(3) How complete are the data? Are they what were specified during the data collection planning phase? Are there any extra data that were not planned for? If so, why are they there?

(4) Most analyses implicitly or explicitly model the data in a parsimonious way and include the possibility of variation from the general description. Each such model suggests its own particular way to identify outliers--the data that deviate remarkably from the general description. Were attempts made to identify and understand outliers at each stage of exploration and analysis?

(5) In many cases it is possible for the analyst to introduce additional data into the analysis for quality checking and insight. For example, many data sets in the natural and social sciences as well as business include (at least implicitly) location information: identifiers of Census regions; names of countries, states, counties; customer zip codes; and so on. Even if--perhaps especially if--spatial correlation is not an element of the EDA or modeling, the analyst can join the data to geographic representations of the locations and map them to look for patterns and outliers.

(6) One of the most insidious errors that can creep into an analysis is loss of data. When extracting fields, summarizing data, reformatting datasets, etc., if one or two items are dropped from a large dataset often there will be nothing to flag it. But occasionally something important is lost, to one's extreme embarrassment if it is ever discovered. Simple checks--such as comparing before and after counts and totals of data--need to occur routinely to guard against such things.

(7) Another insidious error is associated with type conversion in digital computing. For example, recently I had to construct a key (for matching two data files) out of a floating point field. The software (Stata) imported the field as a single precision float in one file but, for whatever reason, as a double precision float in another file. Most of the time the values matched but, in a few cases due to different rounding, they did not. Some data were lost as a result. I caught this only due to the application of (6). In general, it pays to check for consistency of field data types: ints vs. floats, lengths of strings, etc.

(8) If a spreadsheet is ever used at any stage of analysis, expect the worst. The problem is that even a stray keystroke can invisibly corrupt the data. When the results are critical, it pays to keep going back and forth--export to the spreadsheet, do the analysis, import back, and systematically compare--to make sure nothing untoward happened.

(9) Whenever a database is updated, it is worthwhile to pause and perform systematic, complete comparisons with the old one to make sure nothing was lost, changed, or corrupted in the process.

(10) At a higher level, whenever an estimate is performed (such as a regression, PCA, whatever), it can be worthwhile to perform it using a different technique to check for sensitivities or even possible errors in the code. E.g., follow an OLS regression by some form of robust regression and compare the coefficients. For important results, it can be comforting to obtain the answers using two (or more) different software platforms.

Perhaps the best kind of general "consistency check" anyone can perform is to graph everything, early and often.

Answer 2

I suppose this has to do with some form of Quality Control about data integrity, and more specifically that you regularly check that your working database isn't corrupted (due to error during transfer, copy, or after an update or a sanity check). This may also mean ensuring that your intermediate computation are double-checked (either manually or through additional code or macros in your statistical software).

Other information may be found here: the ICH E6 (R1) reference guide about Guideline for Good Clinical Practice from the EMEA, Guidelines on Good Clinical Laboratory Practice, or Clinical Research Study Investigator's Toolbox.