3
$\begingroup$

The title says it all. I'm interested in getting a better understand how to test the predictive power of your statistical models. As an example: Assuming you want to model a certain relationship with regression and test how it behaves to predict future values how do you go about it in a sound statistical sense? How about for more complicated models, like in machine learning or time series analysis? Is there a book with the foundation of such question or at least covering certain parts of it.

$\endgroup$
6
  • $\begingroup$ Testing predictive power is rather specific and simple, so you do not need a whole book on that. You need a test sample on which you will evaluate any model built on a training+validation sample. It is important that the model is built without using any information form the test sample, otherwise your evaluation may not be accurate (may be too optimistic). That is all. $\endgroup$ Mar 17, 2017 at 13:42
  • $\begingroup$ @RichardHardy so cross validation is form a statistical point of you enough? But if there are some nice papers / books still happy to get any references $\endgroup$
    – math
    Mar 17, 2017 at 14:01
  • $\begingroup$ Enough for what? $\endgroup$ Mar 17, 2017 at 14:01
  • $\begingroup$ @RichardHardy enough is maybe the wrong word. So let me try ro rephrase: I used cross validation but can we do something on top of that to get even a better estimate of the predictive power? IMHO, cross validation results can be difficult to be interpreted. $\endgroup$
    – math
    Mar 17, 2017 at 14:04
  • $\begingroup$ Where is the difficulty in interpretation? You get a point estimate, you can get a whole distribution, pretty simple and informative. Also, I would stick to "using a test sample" instead of "cross validation" as the latter is a much broader term. $\endgroup$ Mar 17, 2017 at 14:07

1 Answer 1

1
$\begingroup$

I have heard good things about "The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results"

You get so see how effect size and power are related, a topic that is often seen less than it should be in intro stats books.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.