Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is appreciated.

share|improve this question
add comment

3 Answers

To chl's list, which focuses on frank data processing errors, I would add checks for subtler errors to address the following questions and issues (given in no particular order and certainly incomplete):

  1. Assuming database integrity, are the data reasonable? Do they roughly conform with expectations or conventional models, or would they surprise someone familiar with similar data?

  2. Are the data internally consistent? For example, if one field is supposed to be the sum of two others, is it?

  3. How complete are the data? Are they what were specified during the data collection planning phase? Are there any extra data that were not planned for? If so, why are they there?

  4. Most analyses implicitly or explicitly model the data in a parsimonious way and include the possibility of variation from the general description. Each such model suggests its own particular way to identify outliers--the data that deviate remarkably from the general description. Were attempts made to identify and understand outliers at each stage of exploration and analysis?

  5. In many cases it is possible for the analyst to introduce additional data into the analysis for quality checking and insight. For example, many data sets in the natural and social sciences as well as business include (at least implicitly) location information: identifiers of Census regions; names of countries, states, counties; customer zip codes; and so on. Even if--perhaps especially if--spatial correlation is not an element of the EDA or modeling, the analyst can join the data to geographic representations of the locations and map them to look for patterns and outliers.

  6. One of the most insidious errors that can creep into an analysis is loss of data. When extracting fields, summarizing data, reformatting datasets, etc., if one or two items are dropped from a large dataset often there will be nothing to flag it. But occasionally something important is lost, to one's extreme embarrassment if it is ever discovered. Simple checks--such as comparing before and after counts and totals of data--need to occur routinely to guard against such things.

  7. Another insidious error is associated with type conversion in digital computing. For example, recently I had to construct a key (for matching two data files) out of a floating point field. The software (Stata) imported the field as a single precision float in one file but, for whatever reason, as a double precision float in another file. Most of the time the values matched but, in a few cases due to different rounding, they did not. Some data were lost as a result. I caught this only due to the application of (6). In general, it pays to check for consistency of field data types: ints vs. floats, lengths of strings, etc.

  8. If a spreadsheet is ever used at any stage of analysis, expect the worst. The problem is that even a stray keystroke can invisibly corrupt the data. When the results are critical, it pays to keep going back and forth--export to the spreadsheet, do the analysis, import back, and systematically compare--to make sure nothing untoward happened.

  9. Whenever a database is updated, it is worthwhile to pause and perform systematic, complete comparisons with the old one to make sure nothing was lost, changed, or corrupted in the process.

  10. At a higher level, whenever an estimate is performed (such as a regression, PCA, whatever), it can be worthwhile to perform it using a different technique to check for sensitivities or even possible errors in the code. E.g., follow an OLS regression by some form of robust regression and compare the coefficients. For important results, it can be comforting to obtain the answers using two (or more) different software platforms.

Perhaps the best kind of general "consistency check" anyone can perform is to graph everything, early and often.

share|improve this answer
add comment

I suppose this has to do with some form of Quality Control about data integrity, and more specifically that you regularly check that your working database isn't corrupted (due to error during transfer, copy, or after an update or a sanity check). This may also mean ensuring that your intermediate computation are double-checked (either manually or through additional code or macros in your statistical software).

Other information may be found here: the ICH E6 (R1) reference guide about Guideline for Good Clinical Practice from the EMEA, Guidelines on Good Clinical Laboratory Practice, or Clinical Research Study Investigator's Toolbox.

share|improve this answer
add comment

to add to the other good points

When using Excel, I always generate a case number as the first column for each line, this is then copied to the last column. Excel seems quite happy to sort just a few columns at a time, causing chaos if you are not careful to select them all. Your may not even be aware that this has happened. Being able to check that the case numbers agree in the first and last columns of a line is a useful precaution.

I always review the outliers.

Double entry of data by separate people is recommended for critical work.

When entering data from paper documents, it is a good idea to use a reference identifier to be able to refer back to the exact document and line from which the entry derived, numbering of data entry forms helps with this.

Edit - Another item - I know that editing spreadsheets is fraught with problems, but it is much easier to clean up data entry with them. However, I also keep the original unedited version, so that any changes can be verified or in the worst case restored.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.