I have two very different empirical models that estimate a quantity $Z$ from two quantities ($x$ and $y$). Lets call the two models (highly non linear) $F_1$ and $F_2$

$$ Z = a_1F_1(x,y)$$

$$ Z = a_2F_2(x,y) $$

I trained the model coefficient $a$ over a first dataset (cross-validation) and I am testing on a separate dataset. Model F1 gives smaller absolute error than the other, and also smaller RMSD (root mean square deviation). If I try to overfit $a$ to each realization of Z, the standard deviation of the overfitted $a_1$ is also smaller than for the overfitted $a_2$. To me, that indicates that the process is better described by F1 cause the same coefficient works great for all cases. However, I wonder if I am right to compare the spread (Standard deviation) of coefficients with different units? Should I use some dimensionless quantity such as the Coefficient of Variation (CV)? I see in the Wiki example that has problems when comparing the spread of Celsius versus Faranheit (different units) so I a bit wary. Which is the best option?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.