Should I keep this machine learning resolution? First, happy new year.  And what's my resolution?  I resolve not to peek at my test (out-of-sample) data.  Will I keep it?  Not likely.  The real question is should I?
I get the point of out-of-sample, test sets.   But I cheat a little.  The question is whether my cheating is too much.
Topping the ML commandments is thou shall not use test data more than once.   It's kind of a kosher law not eat your test set or a rule not eat the apple of the test set tree.  Perhaps, we can thank of is as: Thou shall not adulterate one's test set out of marriage.  The test set must remain pure until you've made the sacred commitment and betrothed a final model.  
Actually, I believe there is a new testament ML religion, growing in popularity, that says you don't need to keep kosher; trust in cross-validation (later I explain why I can't trust in CV - but I can still respect you if you do).
Why am I tempted to have pre-marital sex with my test data?  It's simple.  I don't want to end up with a model I can't love.  I spend a lot of time on the ritual of collecting and cleansing data (foreplay?).  Then I build a model.  The ML high priests  preach that I should build lots of models and pick the one that works best.  Then I am to use the test set to estimate the accuracy of my predictions.  I hear it.  I read what I just wrote and it sounds pure.  My problem is that when I build my first model (so far I start with random forests), I'm excited and want to see if it works on the test set just to see if it works at all.  If it works, it motivates me to try other models and to continue.  Without peeking I am left with the worry that all my effort will be in vain.  So I peek. Is this a unforgivable sin (your thoughts appreciated)?
It seems to me that my sin does not condemn me to a bad model (a model with no predictive ability), but that I must reduce my confidence in the accuracy of my predictions
-- A note on my problem with cross-validation. 
I am working with time series; specifically predicting the returns of stocks.  Here is an example of an issue I face with CV.  I've thought about creating 12 folds with each fold representing a month of data (think Jan-Dec) predictors with the subsequent month's return being the response).  It's ok to use Jan-Nov to predict Dec.  But I can't use Feb-Dec to predict Jan.  One of my predictors is past return.  Thus in Mar, the return for Feb is a predictor.  To use that to build a model in which the return for Feb is the response commits the time-series sin of looking-ahead (a mortal sin leading to hellish models that have outstanding in-sample performance and no predictive ability).  Also, within a month, an industry or sector might do well.  A model for a month will typically id an industry or sector predictor.  That will work great in a (within-month) cross-validation, but not nearly as well for a prediction for the future.  Thus cross-validation does not replace an out-of-sample test set in my ML bible.
 A: Yes, you should keep your test set for testing the model and your training set for training it if you are partitioning. 
Cross-validation can and should be applied to the training set during training- it's a way of doing your "peeking" without violating the metaphorical virginity of your test set.
It seems to me that your issues with cross-validation regarding your time series forecast model result from insufficient customization of the cross validation method to the forecast method- i.e. it shouldn't be difficult to imagine a cross-validation which only uses prior data to the forecasted value. 
One such implementation in R is discussed here:
http://robjhyndman.com/hyndsight/tscvexample/
And another has been implemented in the caret package, discussed here:
http://www.r-bloggers.com/time-series-cross-validation-5/
Perhaps implementing a time-series-appropriate method such as these would help to allow you to avoid the temptation to go all the metaphorical way before it is appropriate to do so.
A: You will be overfitting if you were to use your test data to tweak your model. Cross validation is a technique to deal with this problem.
