I recently started my PhD and I am currently working on a project about finding statistical reporting errors. Our work is similar to Nuijten et al. (2016) only for economics.

So, I have a database that consists of statistical tests (like a t-test) gathered from several journals. Additionally, by an easy calculation, I can find if this test might contain an statistical reporting error or not. Since I have several metadata about a test, like the year, the amount of authors, the amount of tests in the article, I want to do a regression that helps me explaining why such errors might occur.

Actually, I have not seen the data yet. That is why I want to do a pre-analysis plan. On of my hypotheses is that the amount of reporting errors gets lower if authors publish in a journal with open data and open code policy.

What would be a good model to check for this hypothesis? I thought about a poisson or a negative binomial model, since the dependent variable should be the amount of statistical tests with a reporting error that are prevalent an article. As exposure variable I could use the amount of tests in a table (the more tests, the more likely to make a mistake). As can be seen in a previous study around 50 % of the articles do not contain an error at all, while for a few articles 26 % of their respectives tests contain errors.

Do you have another idea or would you confirm this model choice? Some other authors do this on a test level by using a logistic regression if a certain tests contain an error or not. Although this seems reasonable, I think on an article level I have more chances for inferences.

Thank you a lot in advance!

  • $\begingroup$ Can you tell us how you will obtain data? $\endgroup$ – kjetil b halvorsen Aug 2 '20 at 0:48
  • 1
    $\begingroup$ Well, not yet unfortunately, since data collection is not yet finished. But I can answer this as soon as we will have a draft! $\endgroup$ – Chris-Gabriel Islam Aug 3 '20 at 14:25

You could fit a binomial regression (a logistic regression where the data is a count of successes over a count of trials). This would allow you to infer how the probability that a test in an article is erroneous changes given the meta-data of the article. This generalizes modeling a single yes/no if the article contains at least one erroneous test, thus preventing you from throwing away information. Further it allows you to reasonably compare articles that contain different amounts of tests, and to control for whether the number of tests changes the probability of a test having an error.

You could pair such a model with a model that represents the distribution of total number of tests featured in an article given the meta data (e.g. a Poisson or negative-binomial regression). This could provide even more insights.

A word of caution: your inferences should be taken with a grain of salt, as your data sounds observational in nature (versus experimental). This is fine, its just important avoid pitfalls like overfitting to data, misinterpreting noise for signal, etc.

  • $\begingroup$ Thank you for your answer! I get the hint with the binomial regression but I read this by Trond Reitan and he suggests not to take ratios but rather counts together with an offset/exposure variable. Additionally, what do you mean by pairing the model with "a model that represents the distribution of total number of tests featured in an article given the meta data"? Meaning, how would you pair it? Maybe a formula would help. $\endgroup$ – Chris-Gabriel Islam Aug 3 '20 at 15:43
  • $\begingroup$ To clarify: the binomial model is a count model with a total count parameter and a probability parameter. As a matter of fact, logistic regression is usually used to model yes/no data which is actually count data where the total count parameter is equal to one i.e. the Bernoulli model which is a special case of the binomial model. My proposal has you model counts directly. $\endgroup$ – David Nelson Aug 3 '20 at 19:04
  • $\begingroup$ The second model I proposed would be a model of the distribution of total counts to accompany your regression. Its unecessary for your stated goal, but could provide additional insights. You would simply fit an unbounded count model e.g. a Poisson, geometric, negative-binimial etc. to the total count. $\endgroup$ – David Nelson Aug 3 '20 at 19:04
  • $\begingroup$ Okay, I understand it better now. But what about the assumption of independent trials for Binomial regression? I can't really argue that tests are independent from each other within an article since the same author is responsible for the tests $\endgroup$ – Chris-Gabriel Islam Aug 4 '20 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.