I'm in the initial stages of brainstorming a way to provide simplified, at-a-glance summaries of scientific papers encapsulated in a few statistics, and possibly a value on a scale based on weighted versions of those statistics in order to provide a quick, approximate visual for the relative likelihood that the data within is correct versus a contrasting paper.
For example, if I tracked the replication status of a result on a scale (from lowest to highest) of "Did Not Replicate, No Replication Attempted, Successfully Replicated", then a person looking at two papers with opposing conclusions would assume that the paper with status "Did Not Replicate" is less likely to be correct than the paper with the status "Successfully Replicated" (or "No Replication Attempted"). The assumption is that, all other metrics being equal, being "higher" on the scale is corresponds to an increased likelihood.
What I'm curious about is, what are more metrics (in experimental design or otherwise) that can be used as a rule of thumb to "rank" the relative likelihood of a paper's results being correct versus another paper on the same subject? Is this even possible?
My proposed metrics so far:
- Replication Status: Did Not Replicate, No Replication Attempted, Successfully Replicated
- Type of Study: Retrospective, Prospective, Randomized Experiment
- (Experiments Only) Blindness: None, Blind, Double Blind
- Population Selection: Non-random, Random
- (Surveys Only) Survey Timeframe: Past, Immediate Past, Real-time
- Sample Size
- Confidence Interval