I'm in the initial stages of brainstorming a way to provide simplified, at-a-glance summaries of scientific papers encapsulated in a few statistics, and possibly a value on a scale based on weighted versions of those statistics in order to provide a quick, approximate visual for the relative likelihood that the data within is correct versus a contrasting paper.

For example, if I tracked the replication status of a result on a scale (from lowest to highest) of "Did Not Replicate, No Replication Attempted, Successfully Replicated", then a person looking at two papers with opposing conclusions would assume that the paper with status "Did Not Replicate" is less likely to be correct than the paper with the status "Successfully Replicated" (or "No Replication Attempted"). The assumption is that, all other metrics being equal, being "higher" on the scale is corresponds to an increased likelihood.

What I'm curious about is, what are more metrics (in experimental design or otherwise) that can be used as a rule of thumb to "rank" the relative likelihood of a paper's results being correct versus another paper on the same subject? Is this even possible?

My proposed metrics so far:

  • Replication Status: Did Not Replicate, No Replication Attempted, Successfully Replicated
  • Type of Study: Retrospective, Prospective, Randomized Experiment
  • (Experiments Only) Blindness: None, Blind, Double Blind
  • Population Selection: Non-random, Random
  • (Surveys Only) Survey Timeframe: Past, Immediate Past, Real-time
  • Sample Size
  • Confidence Interval

What makes a paper "correct" is a very difficult idea to wrap one's mind around. It appears to me that you are aiming to classify a paper/study as "correct" based on various metrics/characteristics.

To look at this concept more in depth, I will provide an example comparison based on some of your proposed metrics so far:

Paper A - Randomized Experiment, Double Blind, Random subject selection, Real-time

Paper B - Randomized Experiment, Double Blind, Random subject selection, Real-time

According to your proposed metrics, these papers would be "ranked" the same. How would you split the tie on which paper is more correct? What if Paper A was written by a more experienced researcher or the data were analyzed by a more seasoned statistician than Paper B, these realities would not be captured by any of the aforementioned metrics. A more appropriate sample size calculation may reveal the latter, but power analyses are not every statisticians' specialty.

Although you can say that Paper A is more "correct" than Paper B, both papers/studies may both be poorly designed and/or analyzed, which may or may not result in replication. As such, I firmly believe that evaluating a given paper as a whole provides far more information about a paper than the sum of its parts (although a poorly organized table or figure can be a major eyesore!), and that makes a metric system such as this quite difficult, if not impossible.

To avoid being a complete debbie downer, I have 2 suggestions for additional metrics:

  1. Appropriate Statistical Assumptions - 1) Assumptions were not mentioned 2) Assumptions were mentioned, but unsure of whether they were met or not 3) Assumptions were met

  2. Areas for Future Study - 1) Authors did not provide avenues for future study 2) Authors provided avenues for future study 3) Authors provided avenues for future study with direction as to how one could pursue these said avenues

I hope this all helps!

  • 1
    $\begingroup$ It is helpful, thanks! I don't expect this process to be able to capture all of the subtleties and hold absolute truth (for instance, theoretically a randomized, double blind experiment on a huge sample size which nonetheless makes a critically flawed assumption could still be less true than a study that scores less in all those metrics). But my hope is that it can be a reliable enough rule of thumb to be used as a surface view tool for people browsing a group of papers on a topic, while allowing them to drill down to more detail if desired.. particularly if two opposing papers score closely. $\endgroup$ – roberttdev Aug 29 '13 at 22:14
  • $\begingroup$ @roberttdev, you definitely have a great start, I assumed that was your goal, but wanted to make sure that you were aware of the vast array of scenarios that may impact your ability to distinguish the correctness of a paper based on metrics, which you seem to be very aware. Your question will likely generate some good discussion. +1 $\endgroup$ – Matt Reichenbach Aug 30 '13 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.