I am using Shao's 1993 article "Linear Model Selection by Cross-Validation" as a starting point for the following cross-validation strategy for a machine learning algorithm (U-net, if you care to know):

  1. NOTE: I am actually generating all of the data in this method using pseudo-random number generators based on assumptions about the statistical properties of the physical environment and method in which it would be generated in real life
  2. Randomly select 15 training examples from a data set of size $n = 40$ $(15 \approx \sqrt[4]{40^3})$
  3. After training is complete, store the average loss on $25 (40 - 15)$ validation/testing examples as a measure of the algorithm performance
  4. Repeat steps 1 and 2 above at least $n = 40$ times, and take the average of the average losses as a measure of the algorithm's performance on potential new data from the same statistical distribution as all of the training and validation data
  5. Repeat the entire process above for multiple configurations of the algorithm (differing numbers of "neurons" or even different methods to declare training complete)
  6. Select the algorithm with the best performance produced in step 3 for each

My concerns with this approach are many, but I have not yet come up with any theoretical justification to reject or ignore it. The 1993 paper shows without question (in my view) the superiority of Monte Carlo cross-validation (MCCV) for linear model selection.

That being said, I am now applying the methods of the paper to non-linear algorithms/models, and I would like to know if anyone is aware of research in this area which would cause me to question the validity of my approach. If possible, please ignore the implications of step 0 above and the use of pseudo-random number generation.

NOTE: This method appears unsuitable for big data because the computation time for each cross-validation replicate could be prohibitive without super-computing resources, so I might be forced to continue using small data sets for training and validation/testing. The 1993 paper requires at least $n$ replicates, and recommends $2n$ if possible, so the time to evaluate performance goes up dramatically as the amount $n$ of data examples grows larger.

  • $\begingroup$ P.S. as I mention in my first comment to this answer, I would be very excited to learn that, for example, all machine learning algorithms are actually linear models of error, because the "weights" used for each "neuron" are scalars, so each computation performed is a linear combination of transformed predictors. $\endgroup$ – brethvoice Mar 20 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.