How to prove two methods are "similar"?

Question

We are testing two imaging machines. One of the machines has been validated numerous times. However, we built the new machine and we need to prove that it is similar/comparable to the validated machine.

So, the machines image a certain tube and determine a physical property. With several image loops, we can find the same physical property, with slight variations, and obtain a standard deviation for the physical property for each machine.

We want to prove the similarity of the new method to the old method.

I have basic understanding of statistics. I tried to solve this by showing a large P value, just to realize that it is not sufficient proof.

Can we publish an article if we say that the two methods give the same order of magnitude results? Is this possible or do we need a clear statistical proof?

Answer 1

Large p-values are easy, if you take small samples. If you try to show this by statistical tests, you need to do power analysis first, determine which difference would be of interest and compute the required sample size to detect that. G*Power is a tool often used for that purpose.

You would want to proove that the new method does not systematically measure larger or smaller values, that the variance of reapeated measurements is not bigger than that of the old method and that the variance is not bigger with small or large results.

That said, there is a graphical method you should consider to implement in your report, in which you can visually examine all these questions: the Mean-Difference-Plot or Bland-Altman-Plot. See https://mran.microsoft.com/web/packages/BlandAltmanLeh/vignettes/Intro.html for an introduction and free software that implements it.