How to evaluate instrumentation accuracy?

Question

(Edited to include systematic error) I'm running some predictive experiments on a quantitative variable. The dataset is made mostly of data coming from sensors, and the outcome variable is the result of a mechanical test. So both Xs and Y are affected by some systematic error: I guess this means I will never have predictions more accurate than the total systematic error involved in process. How can I detect this error, so I can have some threshold for the maximum accuracy for the model? I don't need a perfect measure (that would be included in instruments documentation, currently not available) but at least some degree of reliability for the model.

First thing that came to my mind is to find some observations with a really similar profile, and check if the real Y has some variability.

Any suggestions? some literature about this? Thanks

Answer 1

I would say your guess is correct in that it's impossible to estimate the bias without some information about the true values, e.g. comparing a "gold standard" measurement to what your sensor or mechanical test says. However, if you have multiple sensors measuring the same thing, you could estimate the relative bias between the sensors. The accepted answer here discusses the concept of repeatability, which would be applicable here.

Further: If the measurement uncertainty in $X$ is large relative to the uncertainty in $Y$, consider using a classical errors-in-variables approach, otherwise the model predictions will be further biased. This paper discusses EIV models and introduces a clever way of doing this using Bayesian modeling without the need for Monte Carlo simulations. Good luck!