10
$\begingroup$

I'm interested in ways to detect mistakes in published papers without analyzing the raw data. For example the GRIM test [1]. Here's another similarish one from one of the GRIM authors' blog. I don't know of any others.

Looking for inconsistencies in reported stats seems attractive, because digging through raw data is difficult and sometimes the data isn't available. It's probably also easier to automate.

Edit: Benford's law, credit to DJohnson. Any others?


[1] Brown et al., A Simple Technique Detects Numerous Anomalies in the Reporting of Results in Psychology, Social Psychological and Personality Science (2016)

$\endgroup$
11
  • 4
    $\begingroup$ There are a few blogs that much more systematically track inconsistencies in reported data, e.g., Data Colada (datacolada.org) or Andrew Gelman's blog (andrewgelman.com). Benford's Law is used in accounting to detect nonrandom, i.e., fraudulent, reported numbers. $\endgroup$ Oct 23, 2017 at 18:56
  • 2
    $\begingroup$ Calling them 'tests' is a bit much, we're basically just looking at consistency checks -- which we should all be in the habit of carrying out as we read. They might help you find places to focus attention, but I would be very wary of putting much faith in these things as a way of identifying people doing something wrong (it's trivial to avoid such naive detection methods, so you can only catch the really incompetent). In the example here about the ages -- ...ctd $\endgroup$
    – Glen_b
    Oct 25, 2017 at 21:26
  • 2
    $\begingroup$ ctd... I spotted where the problem was as I was reading the setup. Such errors occur for all manner of innocent reasons and some may not actually be errors (e.g. one person gives their age as 17 years 5 months which was recorded as 17.42 -- there's no error in the reporting at all, though perhaps a slight issue in data-handling). I have many times spotted more egregious things that must have been errors (such as seeing -in a published paper- data grouped into age ranges where the standard deviation reported for one group was considerably larger than half the range of the age-bin). $\endgroup$
    – Glen_b
    Oct 25, 2017 at 21:31
  • 2
    $\begingroup$ There are dozens of things that could easily be checked for. There's a time when I think a set of more comprehensive consistency checks would be useful -- before publication. Where the original data is not provided in the paper, it would be a simple matter for an editor to get someone to run the information in the paper through a collection of consistency checks and flag anything sufficiently weird to query the author about ("With this summary, how do you have a mean equal to the lowest reported value in the group when the standard deviation isn't 0?"). ...ctd $\endgroup$
    – Glen_b
    Oct 25, 2017 at 21:41
  • 2
    $\begingroup$ ctd... That would be handy because if it does turn out to have an innocent explanation that should go in the paper ("Sorry, we forgot to state that the reported means were rounded to the nearest integer." .... "please say so under the table!"). It's fine to do consistency checks and raise queries (if we don't too readily jump to conclusions), but the author there seems to be making rather a big deal out of a fairly obvious thing to check for. $\endgroup$
    – Glen_b
    Oct 25, 2017 at 21:41

2 Answers 2

0
$\begingroup$

You could simply ask for the data if you think there is an error. If they say no that would be a concern to me, although I find it hard to believe people would falsify results deliberately (well some will, but that will be rare I think).

$\endgroup$
0
$\begingroup$

Below are heuristics that are not necessarily direct mistakes, but are frequent ways of using statistics sub-optimally.

Use of underpowered statistical tests, do they mention sample size calculations for example?

Data dredging especially on “big data”. If all variables are highly significant this could be a sign of stepwise regression or similar being used, rather than more logical reasoning having been used.

Over reliance on hypothesis tests, as we as consumers of the paper do not know how many other tests were tried, and compensation schemes for multiple comparisons have side effects such as depending on how many (acknowledged) tests were performed. Further, some fields have much prior knowledge which if not encoded into a test through say a Bayesian approach, are too prone to randomness of “significant” results of hypothesis tests.

Not using multiple imputation or similar but instead dropping observations with missing values from the analysis may bias the remaining data, and also reduces the power of subsequent tests.

Something that is more difficult to discern is if the techniques used are not well understood by the authors. This can be more apparent if they give a presentation of the paper. If the technique is not understood sufficiently, it may have been misused.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.