How would you validate a new test without a gold standard I am measuring a variable (continuous, unknown distribution) on a new assessment tool. There is no gold standard to compare to. I can take 2 measures per participant only. There is no intervention performed between measurements and so ideally, the same value would be returned. I can vary the number of participants. What is the best measure of precision? At this stage, accuracy cannot be assessed but I am just trying to show reproducibility. Can I calculate the number of participants required to do this?
 A: There is no such thing as a gold standard for measurements of that type anyway. All one can do is specify the properties of a reference standard. Validation includes showing experimentally and by simulation that the standard has the properties of a good reference standard. Caution, this is only an outline, a better answer would be book length.
The properties of a good reference standard include, in no particular order

*

*A reference standard is robust. That means that when applied, such a standard does not yield physically ridiculous answers. For example, a one compartment (exponential) plasma concentration-of-drug model is robust, and a two-compartment model, (biexponential mixture model) is not robust as it too frequently yields complex field numerical clearances.

*A reference standard is precise. That means, among other things, that it has the lowest variability for repeat measurements available. For example, a one compartment drug model is imprecise, and more often a two compartment model is more precise, if not robust (1 above).

*A reference standard is unaffected by non-contributory factors. For example a one compartment drug model is affected by fluid overload, and a two compartment model is less affected (but still has decreased clearance for increased fluid load).

*A reference standard is predictive. For example, if data are withheld (for example earlier or later plasma concentrations of drug), that data can be more accurately estimated by extrapolation than with other models. Interpolation should also be accurate.

*A reference standard has physically correct units that balance or agree with what it is purporting to measure. For example, BMI has units of kilograms per square height in meters and is physical nonsense as a measurement of how corpulent someone is whereas body mass to the three fourths power is related to metabolic rate (Kleiber's law) over 23 orders of magnitude of organism size because it has the appropriate fractal geometry.

*A reference standard is explanatory. For example, it better explains the results of divergent experiments and sheds light on how seemingly unrelated experiments should have been conducted or understood to better agree with that standard. This latter goes to accuracy. A reference standard is not only bullet proof, it illuminates everything it touches.

*A reference standard has nice statistical properties, for example, it produces easily understood parameter distributions, often normally distributed, without outliers.

*A reference standard has superior behaviour in binary or round robin controlled trials for the discrimination of subjective and objective utility.

Are there such standards? Read the following papers as example 1, and example 2. The best measure of precision is to do repeat measurement sessions for the same subjects (longitudinal study), provided that there is no interval difference for those measurements, or by showing better detection of interval difference in a longitudinal study. Precision can be compared between methods by processing different time-samples during the same subject test session, see example 3, improved extrapolation testing example 4, validation for imperiousness to extraneous pathological conditions, example 5, better controlled trial results, example 6, better precision and ROC, example 7.

*

*Time Varying Apparent Volume of Distribution and Drug Half-Lives Following Intravenous Bolus Injections, PLoS ONE, 2016

*A gamma-distribution convolution model of 99mTc-MIBI thyroid time-activity curves. EJNMMI Physics, 2016

*Accurate and precise plasma clearance measurement using four 99mTc-DTPA plasma samples over 4 h. Nuclear Medicine Communications, 2016

*The early plasma concentration of 51Cr-EDTA in patients with cirrhosis and ascites: a comparison of three models. Nuclear Medicine Communications, 2016

*Validation of Tikhonov adaptively regularized gamma variate fitting with 24-h plasma clearance in cirrhotic patients with ascites, European Journal of Nuclear Medicine and Molecular Imaging, 2011

*Improved lesion detection from spatially adaptive, minimally complex, Pixon reconstruction of planar scintigraphic images. Computerized Medical Imaging and Graphics, 2005

*A Power Law for Determining Renal Sufficiency Using Volume of Distribution and Weight from Bolus 99mTc-DTPA, Two Blood Sample, Pediatric Data. IEEE Nuclear Science Symposium conference record, 2006

