# Appropriate goodness of fit measure for signal with unknown errors

I have a signal (voltage vs time) from a measurement device. The device outputs exactly one datapoint every constant time interval dt. Theoretical reasons lead me to suppose that the data is following a hyperbolic curve $f(x)= \frac{mx}{k+x}$. Using scipy I performed a fit to the data using the curve_fit function (which does a least squares fit).

I asking myself what would be an appropriate goodness of fit measurement if I don't know the error of the data. Additionally, is it possible to get an estimate of the parameter uncertainty?

I first thought about a Chi-squared test (see my question Chi squared test for goodness of fit ) but there the errors (at least in y) should be known.

If you estimate the curve parameters (i.e. $m$ and $k$) using least squares, then you are implicitly using the root-mean-squared error as the misfit metric (i.e. objective to be minimized).

In general for a regression problem you hypothesize a model of the form $$y=f_\theta(x)+\epsilon$$ where $(x,y)$ is the observed data, $f_\theta$ is a function depending on unknown parameters $\theta$, and $\epsilon$ is an unknown pointwise error (with expected value zero). Generally the parameters $\theta$ are estimated by minimizing some function of the residuals, $r=y-f_\theta(x)$.

In the case of least squares, this is $E(r)=\overline{(r^2)}$, so the "RMSE" corresponds to an estimated standard deviation of the error term (computed over the sampled residuals). If the errors $\epsilon$ are normally distributed, then least squares is a maximum-likelihood estimator of the parameters.

In your case, the errors appear to have some outliers (around $t=0.5$), which will inflate the error-scale estimate (and possibly bias the parameter estimate). You could mitigate this by using a robust estimate for the residual-dispersion (either as a post-processing, or as part of a robust regression scheme). In any case, while a single-number summary is convenient, it is always good to also inspect the residual distribution (both vs. time, and the bulk PDF).

For parameter uncertainty, a simple approach would be bootstrapping. This will also have the benefit of showing the impact of any outliers (as these will be less likely to be included in bootstrap samples).

• I noticed that the errors below a time of 0.2 appear different than the errors above a time of 0.2 - is this an inherent physical characteristic of the measuring device, or is this coincidental to this specific set of measurements? I ask because if it is an inherent physical characteristic of the device you may be able to take advantage of a "low noise region" and "high noise region" in your work. – James Phillips May 29 '17 at 11:19

RMSE (Root Mean Squared Error) might be a good candidate, as historically it was used as a sort of "average magnitude of error". To calculate it, all errors are first squared (to remove any negative values). Those squared values are averaged (the "mean") and then the square root of that mean is taken. In effect it is similar to a "average magnitude of error".

As I understand it, the historical reason that the average magnitude of error was not directly used originally was that "absolute value" was not a differentiable function and as such was an intellectual no-no in the early days - that is, "absolute value" could not be used in the reasoning of calculus. Whatever its origins, RMSE is a widely used and widely understood fit statistic.