A measure like mean square error appears to tolerate some variation in dataset size but, for example, a two-parameter model is still likely to produce a great (yet meaningless) fit to any dataset with less than three points. Are there any standard recommended ways to compare how different datasets (of unequal sizes) fit a model?

Is it generally invalid to compare goodness of fit between different datasets? Is it sufficient to just exclude datasets having fewer data-points than the exact number of free parameters in the model, or should there also be some correction applied to the measure (to favour a very large dataset, with a good RMSE, over a moderate-sized dataset with a slightly better RMSE)? Is there any simple information-theoretical answer (to say which dataset lends the best evidence for the model), or is it always necessary to additionally have a residual-model for the noise or error (and to use that to calculate probabilities of the deviation between each dataset and the family of curves, perhaps needing a full Bayesian approach)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.