Testing methodology I recently built a simple feed forward NN to predict daily demand (48 output neutrons, representing half hours) based upon 32 input features. I tested the performance by firstly doing 10 fold cross validation and after the network has been trained I used a test set to get a final ‘production scenario’ metric.
I am very happy with the performance and want to push the network into production. Now, I get these 32 features daily and the idea is to predict the current day and retrain the network daily/ weekly autonomously. My question is, is it sensible to retrain using only cross validation without a test set or is there a risk of overfitting? The network is retrained with completely original architecture with the same hyper parameters and this fact makes me think that I might not need a test set as I previously confirmed there is no overfitting and/or other problems. But I am not entirely confident my thinking is correct and would love to hear other people’s thoughts. 
 A: It's very common to do it the other way around - that is to say, to train a final model using the full training set without CV, while continuing to keep the test set separate. When doing it this way, cross validation is not explicitly used for the final model but rather the hyperparameters are taken from the earlier cross validation. The rationale behind this is that each model trained during CV was trained on only $\frac{k-1}{k}$ of the data, say 80% for 5-fold CV, and a model trained on 100% will be ever so slightly better, while the hyper-parameters discovered from CV are likely to still be optimal if we bump the data size up a little bit. 
However, I don't believe I've ever seen anyone advocate also including the hold-out test set into the mix when training this final model, and for good reason. The Elements of Statistical Learning, "Chapter 7: Model Selection and Assessment", has the following diagram:

Where the idea is that both "train" and "validation" sets are used to build a model, while "test" is only ever used to evaluate a model. The difference between "train" and "validation" is that the training set is fed into some kind of optimization process to find optimal parameters, which the validation set is used to choose hyper-parameters. In cross validation, of course, the validation set moves around and is always the fold we don't optimize on.
Because terminology is not 100% standardized, and some people use "test set" to be "validation set" or even transpose the terminology, I always make it a point at work to refer to the test set as the "hold-out test set." Adding the phrase "hold-out" emphasizes that we will never train on it, nor use it in any way to select or tune the model. 
Without a proper hold-out test set, you will have no way to report on the model's performance without having to add a caveat about overfitting. For example, you couldn't simply say, "our new model achieves 0.05 higher AUC compared to prior work" without also adding "I assume, but don't really know; I measured it in such a way that gave my model an unfair advantage, so for all I know it could be the same or worse." In either an academic or business context, that is not a statement that is likely to received with unmitigated enthusiasm. 
