First, happy new year. And what's my resolution? I resolve not to peek at my test (out-of-sample) data. Will I keep it? Not likely. The real question is should I?
I get the point of out-of-sample, test sets. But I cheat a little. The question is whether my cheating is too much.
Topping the ML commandments is thou shall not use test data more than once. It's kind of a kosher law not eat your test set or a rule not eat the apple of the test set tree. Perhaps, we can thank of is as: Thou shall not adulterate one's test set out of marriage. The test set must remain pure until you've made the sacred commitment and betrothed a final model.
Actually, I believe there is a new testament ML religion, growing in popularity, that says you don't need to keep kosher; trust in cross-validation (later I explain why I can't trust in CV - but I can still respect you if you do).
Why am I tempted to have pre-marital sex with my test data? It's simple. I don't want to end up with a model I can't love. I spend a lot of time on the ritual of collecting and cleansing data (foreplay?). Then I build a model. The ML high priests preach that I should build lots of models and pick the one that works best. Then I am to use the test set to estimate the accuracy of my predictions. I hear it. I read what I just wrote and it sounds pure. My problem is that when I build my first model (so far I start with random forests), I'm excited and want to see if it works on the test set just to see if it works at all. If it works, it motivates me to try other models and to continue. Without peeking I am left with the worry that all my effort will be in vain. So I peek. Is this a unforgivable sin (your thoughts appreciated)?
It seems to me that my sin does not condemn me to a bad model (a model with no predictive ability), but that I must reduce my confidence in the accuracy of my predictions
-- A note on my problem with cross-validation. I am working with time series; specifically predicting the returns of stocks. Here is an example of an issue I face with CV. I've thought about creating 12 folds with each fold representing a month of data (think Jan-Dec) predictors with the subsequent month's return being the response). It's ok to use Jan-Nov to predict Dec. But I can't use Feb-Dec to predict Jan. One of my predictors is past return. Thus in Mar, the return for Feb is a predictor. To use that to build a model in which the return for Feb is the response commits the time-series sin of looking-ahead (a mortal sin leading to hellish models that have outstanding in-sample performance and no predictive ability). Also, within a month, an industry or sector might do well. A model for a month will typically id an industry or sector predictor. That will work great in a (within-month) cross-validation, but not nearly as well for a prediction for the future. Thus cross-validation does not replace an out-of-sample test set in my ML bible.
