Appropriate model for amount of statistical errors in articles I recently started my PhD and I am currently working on a project about finding statistical reporting errors. Our work is similar to Nuijten et al. (2016) only for economics.
So, I have a database that consists of statistical tests (like a t-test) gathered from several journals. Additionally, by an easy calculation, I can find if this test might contain an statistical reporting error or not. Since I have several metadata about a test, like the year, the amount of authors, the amount of tests in the article, I want to do a regression that helps me explaining why such errors might occur.
Actually, I have not seen the data yet. That is why I want to do a pre-analysis plan. On of my hypotheses is that the amount of reporting errors gets lower if authors publish in a journal with open data and open code policy.
What would be a good model to check for this hypothesis? I thought about a poisson or a negative binomial model, since the dependent variable should be the amount of statistical tests with a reporting error that are prevalent an article. As exposure variable I could use the amount of tests in a table (the more tests, the more likely to make a mistake).
As can be seen in a previous study around 50 % of the articles do not contain an error at all, while for a few articles 26 % of their respectives tests contain errors.
Do you have another idea or would you confirm this model choice? Some other authors do this on a test level by using a logistic regression if a certain tests contain an error or not. Although this seems reasonable, I think on an article level I have more chances for inferences.
Thank you a lot in advance!
 A: You could fit a binomial regression (a logistic regression where the data is a count of successes over a count of trials). This would allow you to infer how the probability that a test in an article is erroneous changes given the meta-data of the article. This generalizes modeling a single yes/no if the article contains at least one erroneous test, thus preventing you from throwing away information. Further it allows you to reasonably compare articles that contain different amounts of tests, and to control for whether the number of tests changes the probability of a test having an error.
You could pair such a model with a model that represents the distribution of total number of tests featured in an article given the meta data (e.g. a Poisson or negative-binomial regression). This could provide even more insights.
A word of caution: your inferences should be taken with a grain of salt, as your data sounds observational in nature (versus experimental). This is fine, its just important avoid pitfalls like overfitting to data, misinterpreting noise for signal, etc.
