2
$\begingroup$

In order to evaluate and compare count models (e.g. Poisson regression), we can calculate scoring rules (e.g. Brier Score, Dawid-Sebastiani score, etc.) which are explained here: Error metrics for cross-validating Poisson models.

Should we calculate these scores using the data used for estimating the models (training data) or on a data subset that the models have not seen before (validation data)? Does doing the former lead to choosing models that are over-fitting and less generalizable? Is over-fitting necessarily a bad thing, if we are using the model only for inference?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If you are using your model for inference, then you should either pre-specify the model completely (i.e., not use any model selection, whether based on a scoring rule, information criterion or other), or you need to correct your p values for the model selection step. Otherwise your p values will be biased low, since your selected model will contain predictors that are "useful", which means that their coefficients will be biased towards significance.

If you do so correct, then you can use scoring rules in-sample for model selection. (However, I do not know of any literature exploring p value correction for scoring rule based model selection.)

Of course, if you want to use your model for prediction, it makes more sense to evaluate your scoring rules on test data.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ There is no theory for what I am modeling, and I had a large number of variables (35ish) to choose from to include in the model. I used all-subset variable selection to minimize AICc (all subsets were 2^35 = 34*10^9, and impractical, so I used a genetic algorithm through the R package glmulti). I am using the final subset of explanatory variables, with different model specifications (e.g. different types of random effects specification, spatial vs. non-spatial, etc.). To choose among those models, I am trying to use scoring rules. Do you think I still need p-value correction? $\endgroup$
    – Fred
    May 20, 2019 at 14:19
  • 1
    $\begingroup$ Indeed I think you do. Have you tried simulating data with zero predictor effects whatsoever and putting this through your workflow? Do this multiple times, with different RNG seeds. I strongly suspect you will get quite a number of "significant" effects. $\endgroup$
    – Stephan Kolassa
    May 20, 2019 at 14:32
  • 1
    $\begingroup$ Unfortunately, no. It's a difficult problem. A search on CV does not really help. ... $\endgroup$
    – Stephan Kolassa
    May 20, 2019 at 15:20
  • 1
    $\begingroup$ ... A permutation test might be your best bet: do the procedure I outlined 10,000 times and note the parameter estimates for all predictors (predictors that drop out in one run get a zero parameter). You now have a distribution of the parameter estimates under the null hypothesis of no relationship which takes your model building procedure into account. ... $\endgroup$
    – Stephan Kolassa
    May 20, 2019 at 15:22
  • 1
    $\begingroup$ ... Now run your procedure on your original dataset. All parameter estimates that fall in the tails of the permutation based distributions are significant. The drawback: you need to run your model selection algorithm 10,000 times, which may be impossible if it takes a couple of minutes. (This is trivially parallelizable, though.) There are a couple of relevant books by Good, Resampling Methods and Permutation, Parametric, and Bootstrap Tests of Hypotheses. $\endgroup$
    – Stephan Kolassa
    May 20, 2019 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.