TLDR: You can see my neural network learning curves here: http://imgur.com/0CL6LVY. Which regularization term would you pick given that the test error actually drops below the training error at some point. See implementation details below.
I just trained a neural network with 33 input units, 10 hidden units, and a single output unit. The hidden units each use a sigmoid activation function and the final output is just a linear combination of those. No activation function is used for the output because the goal is regression rather than classification.
I trained the network using backpropagation and some annealing at the end to find a better local minima. I used a training set of 7800 reserving 2600 for testing the error and 2600 for cross-validation. The error term I minimized was the mean squared error of the network output with the actual output.
I trained the network in several passes using different l2 regularization constants and I plotted the learning curves here: http://imgur.com/0CL6LVY. As expected the training error grows with the regularization term and the training/cv error shrinks. My question is, there is a point when the training error and test error cross and then the testing error becomes and stays smaller than the training error. In examples I've seen there is an ideal regularization term that minimizes the test set error but it still lies above the training error. What would cause the test set error to become so low? Also, which regularization term would you pick? The one where the test/training error cross or a larger term that further minimized the test set error. Thanks.