3
$\begingroup$

The title says it all. I'm interested in getting a better understand how to test the predictive power of your statistical models. As an example: Assuming you want to model a certain relationship with regression and test how it behaves to predict future values how do you go about it in a sound statistical sense? How about for more complicated models, like in machine learning or time series analysis? Is there a book with the foundation of such question or at least covering certain parts of it.

$\endgroup$
  • $\begingroup$ Testing predictive power is rather specific and simple, so you do not need a whole book on that. You need a test sample on which you will evaluate any model built on a training+validation sample. It is important that the model is built without using any information form the test sample, otherwise your evaluation may not be accurate (may be too optimistic). That is all. $\endgroup$ – Richard Hardy Mar 17 '17 at 13:42
  • $\begingroup$ @RichardHardy so cross validation is form a statistical point of you enough? But if there are some nice papers / books still happy to get any references $\endgroup$ – math Mar 17 '17 at 14:01
  • $\begingroup$ Enough for what? $\endgroup$ – Richard Hardy Mar 17 '17 at 14:01
  • $\begingroup$ @RichardHardy enough is maybe the wrong word. So let me try ro rephrase: I used cross validation but can we do something on top of that to get even a better estimate of the predictive power? IMHO, cross validation results can be difficult to be interpreted. $\endgroup$ – math Mar 17 '17 at 14:04
  • $\begingroup$ Where is the difficulty in interpretation? You get a point estimate, you can get a whole distribution, pretty simple and informative. Also, I would stick to "using a test sample" instead of "cross validation" as the latter is a much broader term. $\endgroup$ – Richard Hardy Mar 17 '17 at 14:07
0
$\begingroup$

I have heard good things about "The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results"

You get so see how effect size and power are related, a topic that is often seen less than it should be in intro stats books.

$\endgroup$
0
$\begingroup$

This video covers a lot on this https://www.youtube.com/watch?v=85dtiMz9tSo&t=169s

$\endgroup$
  • $\begingroup$ Since the question is a reference request, you don't need to explain the full content of your link here, but your suggested reference would be much more useful to readers if you could include more details. For example: what aspects of the original question are covered in your reference, the manner in which it is presented, the expected level of prior knowledge and so on. (The original question did ask for a book, but an appropriate video reference may still have some relevance.) $\endgroup$ – Silverfish Mar 17 '17 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.