I have a novel model for the errors that affect many types of qubits (quantum bits) and want to show my theory is correct. Visually it is great, but that is not quantitative.
I'm a theorist, while my colleagues testing my ideas are experimentalists. They are happy using certain statistical methods that use assumptions which are not true. I would like to make a statement about my fitted variables that is rigorous.
I solve a sophisticated nonlinear system of ODEs and compare a measurement of the system and the data they found. There may be systemic errors that we don't account for. Also, their measurement errors may not be normal and independent.
The figure shows a typical correspondence between my model and the experimental data. I optimize the model concurrently for 7 such plots and repeat for roughly 20 such experiments. All visually agree well. There are roughly 300 data points (each with an associated error estimate) and at most 4 parameters that I fit.
I use a weighted least squares method (LsqFit.jl) to fit the model and find reduced chi2 values between 0.96 and 1.67. However, when is a chi2 sufficiently small? The weighted least squares method also produces means of the fit variables and error estimates using the estimated covariance matrix. The error estimates are hard to believe and inconsistent. How can I better find confidence intervals? Is there a quantity better than reduced chi2 to report?
I apologize in advance, statistics is not my area of expertise.
