What data qualities could one use to evaluate the likelihood of a paper's result being correct? I'm in the initial stages of brainstorming a way to provide simplified, at-a-glance summaries of scientific papers encapsulated in a few statistics, and possibly a value on a scale based on weighted versions of those statistics in order to provide a quick, approximate visual for the relative likelihood that the data within is correct versus a contrasting paper.
For example, if I tracked the replication status of a result on a scale (from lowest to highest) of "Did Not Replicate, No Replication Attempted, Successfully Replicated", then a person looking at two papers with opposing conclusions would assume that the paper with status "Did Not Replicate" is less likely to be correct than the paper with the status "Successfully Replicated" (or "No Replication Attempted").  The assumption is that, all other metrics being equal, being "higher" on the scale is corresponds to an increased likelihood.
What I'm curious about is, what are more metrics (in experimental design or otherwise) that can be used as a rule of thumb to "rank" the relative likelihood of a paper's results being correct versus another paper on the same subject?  Is this even possible?
My proposed metrics so far:


*

*Replication Status: Did Not Replicate, No Replication Attempted, Successfully Replicated

*Type of Study: Retrospective, Prospective, Randomized Experiment

*(Experiments Only) Blindness: None, Blind, Double Blind

*Population Selection: Non-random, Random

*(Surveys Only) Survey Timeframe: Past, Immediate Past, Real-time

*Sample Size

*Confidence Interval
 A: What makes a paper "correct" is a very difficult idea to wrap one's mind around. It appears to me that you are aiming to classify a paper/study as "correct" based on various metrics/characteristics. 
To look at this concept more in depth, I will provide an example comparison based on some of your proposed metrics so far:
Paper A - Randomized Experiment, Double Blind, Random subject selection, Real-time
Paper B - Randomized Experiment, Double Blind, Random subject selection, Real-time
According to your proposed metrics, these papers would be "ranked" the same. How would you split the tie on which paper is more correct? What if Paper A was written by a more experienced researcher or the data were analyzed by a more seasoned statistician than Paper B, these realities would not be captured by any of the aforementioned metrics. A more appropriate sample size calculation may reveal the latter, but power analyses are not every statisticians' specialty.
Although you can say that Paper A is more "correct" than Paper B, both papers/studies may both be poorly designed and/or analyzed, which may or may not result in replication. As such, I firmly believe that evaluating a given paper as a whole provides far more information about a paper than the sum of its parts (although a poorly organized table or figure can be a major eyesore!), and that makes a metric system such as this quite difficult, if not impossible.
To avoid being a complete debbie downer, I have 2 suggestions for additional metrics:


*

*Appropriate Statistical Assumptions - 1) Assumptions were not mentioned 2) Assumptions were mentioned, but unsure of whether they were met or not 3) Assumptions were met

*Areas for Future Study - 1) Authors did not provide avenues for future study 2) Authors provided avenues for future study 3) Authors provided avenues for future study with direction as to how one could pursue these said avenues
I hope this all helps!
