In order to evaluate and compare count models (e.g. Poisson regression), we can calculate scoring rules (e.g. Brier Score, Dawid-Sebastiani score, etc.) which are explained here: Error metrics for cross-validating Poisson models.
Should we calculate these scores using the data used for estimating the models (training data) or on a data subset that the models have not seen before (validation data)? Does doing the former lead to choosing models that are over-fitting and less generalizable? Is over-fitting necessarily a bad thing, if we are using the model only for inference?