2
$\begingroup$

A measure like mean square error appears to tolerate some variation in dataset size but, for example, a two-parameter model is still likely to produce a great (yet meaningless) fit to any dataset with less than three points. Are there any standard recommended ways to compare how different datasets (of unequal sizes) fit a model?

Is it generally invalid to compare goodness of fit between different datasets? Is it sufficient to just exclude datasets having fewer data-points than the exact number of free parameters in the model, or should there also be some correction applied to the measure (to favour a very large dataset, with a good RMSE, over a moderate-sized dataset with a slightly better RMSE)? Is there any simple information-theoretical answer (to say which dataset lends the best evidence for the model), or is it always necessary to additionally have a residual-model for the noise or error (and to use that to calculate probabilities of the deviation between each dataset and the family of curves, perhaps needing a full Bayesian approach)?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.