9
$\begingroup$

I have some predictive models whose performance I would like to back-test (i.e., take my dataset, "rewind" it to a previous point in time, and see how the model would have performed prospectively).

The problem is that some of my models were built via an interactive process. For instance, following the advice in Frank Harrell's Regression Modeling Strategies, in one model I used restricted cubic splines to handle possible nonlinear associations between features and the response. I allocated the degrees of freedom of each spline based on a combination of domain knowledge and univariate measures of strength of association. But the degrees of freedom that I want to allow my model obviously depends on the size of the dataset, which varies dramatically when backtesting. If I don't want to hand-pick degrees of freedom separately for each time at which the model is backtested, what are my other options?

For another example, I'm currently working on outlier detection via finding points with high leverage. If I were happy to do this by hand, I would simply look at each high-leverage data point, sanity-check that the data was clean, and either filter it out or clean it up by hand. But this relies on a bunch of domain knowledge, so I don't know how to automate the process.

I would appreciate advice and solutions both (a) to the general problem of automating interactive parts of the model-building process, or (b) specific advice for these two cases. Thanks!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
+50
$\begingroup$

FYI, this might be more appropriate for SE.DataScience, but for the time being, I'll answer it here.

It seems to me like you might be in a situation where you will have no choice but to write a script that will implement your solutions. Never having worked with splines, my knowledge of them is strictly theoretical so please bear with me and let me know if there is anything I'm not seeing.

Broadly speaking, it appears that you have a couple of different items that you will have to resolve in order to implement this.

1.) Determining the model parameters in a dynamic fashion. You have previously mentioned that you've used a combination of domain knowledge and univariate measures. That seems to me like something that you should be able to handle heuristically. You will have to agree at the outset on a set of rules which your program will implement. This may or may not be a trivial task as you will have to do some hard thinking about the potential implications of those rules. This may require you to re-visit every step of your process and cataloging not just the decisions, but also the reasons behind those decisions.

2.) Actually implementing your program. In order to make your performance testing properly dynamic and easy to maintain and modify going forward, you will have to think about how you're going to structure it. You will likely want to use some sort of loop for your main model predictive performance estimation, preferably with a user-definable length in order to allow for greater flexibility going forward. You will also likely want to write separate functions for each action that you want your program to take as this will make it easier to test functionality, and to maintain and modify your program going forward. You will, at a minimum, likely need functions for dataset selection (i.e. only time periods that have "gone by" at the moment of backtesting), cleaning and validation (which you'll really have to think about, as data munging is a critical part of model building), functions for model training parameters, and functions for model prediction and performance measure collection and storage.

Your question about outlier detection and handling also falls under those two concerns and I would go about implementing by writing smaller loops within your main program loop that would continue to "clean" and refit the model until it's reached a point where you would be happy with it (which again, you'll have to define yourself).

If this sounds like a big task, it's because it is; people have written entire software libraries (sometimes very lucratively) in order to perform this sort of task. Beyond that, it's hard to offer any more specific advice without knowing more about your processes, data structure, and the programming language you've done your work in thus far.

If any of this of useful to you and you'd like me to expand on any of it, comment, let me know, and I'd be more than happy to do so.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I don't need any help actually writing the code, thanks--our backtesting infrastructure is already in place and quite strong. I'm just interested in what statistical procedures one might use. With regard to heuristically automating the interactive part of model-building: has anything been written about this? I haven't seen any mentions of this kind of process in the literature. You mention "people have written entire software libraries"--do you have any references? $\endgroup$
    – Ben Kuhn
    May 7, 2015 at 1:40
  • $\begingroup$ @BenKuhn - Based on your comment, I'm a little unclear on the exact difficulties you're having; please help me get a little more clarity. The use of heuristics in automated model building is quite widespread; the most basic application I can think of right now is the humble stepwise regression. Lacking the exact details of your model, I can't point to the exact pieces of literature that might help you, but a cursory Google search brings up several articles exploring methods for automatic parameter selection, particularly for smoothing and penalized splines. See my next comment for a few links $\endgroup$
    – habu
    May 7, 2015 at 10:09
  • $\begingroup$ link.springer.com/article/10.1023/A:1008975231866, onlinelibrary.wiley.com/doi/10.1111/j.1467-842X.2007.00473.x/…, biomet.oxfordjournals.org/content/98/1/225.short $\endgroup$
    – habu
    May 7, 2015 at 10:10
  • $\begingroup$ @BenKuhn - what specifically do you mean when you say statistical procedures you might use? To my mind, the backtest could be handled fairly straightforward by using train-test sampling with either a rolling or expanding window of data selection. All the data you've acquired up to the point of the backtest would be your training set, while the data you would expect to see in the next time period, before you have a chance to readjust your model, would be your test set. All the usual measures of predictive performance and goodness of fit could be used to perform the actual evaluation. $\endgroup$
    – habu
    May 7, 2015 at 10:20
  • $\begingroup$ @BenKuhn - Implementing the actual business knowledge portion would require you to codify it and ensure that the data needed to make such determinations is available when necessary. Also, I use the term "software library" as a blanket term covering everything from extensions to existing modeling libraries that are meant to automate model building for certain applications, all the way to industrial-grade, proprietary expert and decision support systems. $\endgroup$
    – habu
    May 7, 2015 at 10:47
3
$\begingroup$

Rather than trying to figure out how to automate your manual model tuning efforts, I would circumvent that problem all together by looking into lower variance learners that require far less tuning, even if that is at some cost of increased model bias. You want confidence in your backtest results which largely comes down to low sampling variance in your predictions, and introducing some automated tuning process on top of a learner that already has sampling variance itself is working against that goal. It might seem like the tail is wagging the dog here, but anything that requires a lot of careful tuning (manual or automated) is not a great candidate for a truly honest backtest environment IMO.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Why wouldn't automated tuning (with a separate tuning run at each backtest time point) be a "truly honest backtest environment"? $\endgroup$
    – Ben Kuhn
    May 7, 2015 at 1:41
  • $\begingroup$ Reducing variance by dropping the splines would lead to an unacceptable loss in predictive power, unfortunately. Is that what you were thinking of when you suggested to use a lower-variance learner? If not, what were you thinking of? $\endgroup$
    – Ben Kuhn
    May 7, 2015 at 1:46
  • $\begingroup$ @BenKuhn - I share andrew's concerns about whether a backtest would be a truly "honest" test of the model's out-of-sample predictive power, if for no other reason than the fact that it appears that you have developed your tuning parameters on the entire dataset available to you; even if you "wind the clock back" and rebuild your model dynamically, the methodology by which you will be doing so will have been developed by reference to the entire dataset, so the risk exists that the model will still overfit, even if it is re-trained on a subset of the available data. $\endgroup$
    – habu
    May 7, 2015 at 10:56
  • 1
    $\begingroup$ To clarify, automated tuning would make it honest in the sense that predictions at time $t$ don't depend on data from time greater than $t$. To @habu's point, there is always going to be some irreducible amount of in-sample bias that results from iteratively improving your model based on backtest performance, and I don't really see a way around that (I'm assuming this is a finance application). The point I was making was that your confidence interval about your backtest results is related to the sampling variance of the predictor and the tuning process on top of it. $\endgroup$
    – andrew
    May 7, 2015 at 14:39
  • 1
    $\begingroup$ And in a domain as noisy as finance, you want make sure that if history had unfolded a bit differently (but still drawn from some underlying distribution) you would still arrive at a similar model. If you are confident that your process is robust to sampling variance than I think you're good. But in my experience automated tuning procedures can be very sensitive to sampling variance. $\endgroup$
    – andrew
    May 7, 2015 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.