I have a 6100 sample data set (randomly split 60-20-20 between training, validation, and test subsets) with 8-12 features. I'd like to know if I should be concerned about any of the following weird observations as I conduct multivariate linear regression, polynomial regression, and a 3 layer neural network:
- Validation set consistently ends up with a smaller error than the training set's (see attached image).
- The test set's error, despite large sample size, seems to settle in the large range of 5-8 (one time it even came back at 13) depending on how the sample is randomly split. I would've expected a much tighter and consistent range, and a difference in the range of error between Neural Networks and the regressions.
- The test set's error, with enough iterations, will often come in BELOW that of the validation set's error.
- Regardless of whether I use multivariate linear regression, polynomial regression, or the neural network, and regardless of the test set's error, the optimized hypotheses all seem to output similar predictions- eg. about 83% of the sample's output is predicted within 1 standard deviation of the actual result, regardless of whether the error was on the large side or not. I would've expected better accuracy to correspond with smaller error, and smaller error to correspond with the more sophisticated models (polynomial regression and neural networks over linear regression).