I have a 6100 sample data set (randomly split 60-20-20 between training, validation, and test subsets) with 8-12 features. I'd like to know if I should be concerned about any of the following weird observations as I conduct multivariate linear regression, polynomial regression, and a 3 layer neural network:

  • Validation set consistently ends up with a smaller error than the training set's (see attached image).
  • The test set's error, despite large sample size, seems to settle in the large range of 5-8 (one time it even came back at 13) depending on how the sample is randomly split. I would've expected a much tighter and consistent range, and a difference in the range of error between Neural Networks and the regressions.
  • The test set's error, with enough iterations, will often come in BELOW that of the validation set's error.
  • Regardless of whether I use multivariate linear regression, polynomial regression, or the neural network, and regardless of the test set's error, the optimized hypotheses all seem to output similar predictions- eg. about 83% of the sample's output is predicted within 1 standard deviation of the actual result, regardless of whether the error was on the large side or not. I would've expected better accuracy to correspond with smaller error, and smaller error to correspond with the more sophisticated models (polynomial regression and neural networks over linear regression).

I'm using Octave code successfully used in Andrew Ng's online ML course exercises. Learning Curves


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.