0
$\begingroup$

As a part of my research I create an explanatory negative binomial regression model. Now, I want to show this model can also have predictability power. I don't want to compare my model with other competing models. I just want to show that the model has some predictability power. I would be really thankful if you can help me on that.

$\endgroup$
  • 1
    $\begingroup$ I was going through my old answers and noticed this one was not accepted. Do you perhaps need further clarification? $\endgroup$ – Richard Hardy Feb 19 '17 at 11:25
1
$\begingroup$

Once you have used your whole sample for model selection and fitting, you cannot test in a fair way how well the model can predict. If you test on the same data that was used for model building, you will get overly optimistic results as the fitted model has adapted to that particular data set to fit it well. You will need a new sample for a fair evaluation.

Alternatively, you could go back and redo your analysis in the following way (which might ultimately lead to selecting a different model than you have now).

  • Split your sample into training, testing and validation parts (subsamples).
  • Keep the validation part hidden until you have chosen and fitted your model on the training and testing parts.
  • Try your chosen model on the validation part and evaluate its performance there. That would be a fair evaluation.
  • Refit the model on the whole data set before applying it elsewhere.

If you explicitly refuse to compare your model with other models and do not want to do model selection (e.g. you have a model entirely implied by theory or somehow imposed on you), you could skip model selection using training and testing subsets; but you still need to separate the data for model fitting (given model specification) and model validation to get a fair evaluation of forecasting performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.