Training a neural network with a training set with no noise

Question

I am using artificial neural networks for an unconventional problem and, although it looks like it is working, I want to make sure that I am not doing anything wrong.

I have a code, STARCODE, that predicts the spectrum of stars based on some their physical properties (input parameters). STARCODE solves many physical equations and it takes several hours to run it. At the end of this process, it yields the corresponding spectrum of that star. I want to use STARCODE to fit a large sample of observed spectra, but the long time required to compute each model prevents me from using statistical methods such as MCMC. My idea is to train a neural network that receives the same input as STARCODE, and outputs a spectrum similar to what STARCODE would yield if ran. STARCODE is deterministic, i.e., inputing the same parameters will always yield the same output spectrum.

I have a considerable number of pre-computed STARCODE models that I can use to train this neural network. I had originally split these precomputed models into training and validation samples and trained the neural network in a standard way, checking the mean squared error of the validation set to ensure that I do not overtrain. However, when I looked at the MSE of the validation set as a function of training iteration, I found out that it just gets at a minimum value and stays around it (see figure), instead of increasing again as one would expect once overtraining occurs.

The resulting neural network does a great job predicting the output spectra of STARCODE, i.e., when given the same input parameters, the real STARCODE output and the prediction from the neural network are very similar (enough for my purposes). Using a third set of STARCODE spectra that were not used for training or validating, there is no sign of overtraining: the artificial neural network predicts them correctly too. I want to confirm a couple of ideas before I start using the neural network for the analysis:

• I think the MLE behavior arises from the fact that these models do not have any noise in them. Is this correct? If so, does it mean that it is not possible to overtrain the neural network in this particular case?

• If that is also correct, I guess the most optimal way to train the neural network would be to:

  1) determine the number of neurons by increasing it until the final MSE value does not improve anymore (something that I am doing already), and then

  2) once I have estimated the optimal number of neurons, I do not need to use a validation set since, if I am right, the neural network cannot be overtrained.

Is my interpretation of the MSE behavior correct, or am I missing something?

Answer 1

Your MSE plot is really supposed to also contain a line for the validation MSE and testing MSE. The MSE of the validation may go up -- as you eluded to. For continuous input features, sometimes when using an ANN for function approximation, e.g. fitting a 3D surface curve, the spacing of the observed data can cause the ANN to literally bomb out (totally fail), so the workaround is to employ Latin Hypercube Sampling (LHS) when the input is essentially a multi-dimensional grid. You mentioned input spectra, so it sounds like your input is maybe an energy-steradian grid of some sort. (so it sounds like you are not using the ANN to classify discrete objects, but rather predict using all continuous data).

Other than the above issues, other things you might consider is use of "cross-entropy" instead of MSE -- so see Bishop's classic ANN book available free online. Since many of your input spectra result in a matrix essentially being fed to the ANN, try using PCA to reduce dimensions and accept dimensions whose eigenvalues $\lambda_j>1$ as a start. That way you will prevent correlated data from being used in the ANN. Correlated data slow down an ANN's learning, since it wastes time learning the correlation. Orthogonality is the best thing to feed an ANN. FYI - PCA is also a very common technique to denoise data for input into an ANN.

Below are images for ANN MSE with and without use of PCA.

Without PCA feature processing:

With PCA feature processing: