I am using artificial neural networks for an unconventional problem and, although it looks like it is working, I want to make sure that I am not doing anything wrong.
I have a code, STARCODE, that predicts the spectrum of stars based on some their physical properties (input parameters). STARCODE solves many physical equations and it takes several hours to run it. At the end of this process, it yields the corresponding spectrum of that star. I want to use STARCODE to fit a large sample of observed spectra, but the long time required to compute each model prevents me from using statistical methods such as MCMC. My idea is to train a neural network that receives the same input as STARCODE, and outputs a spectrum similar to what STARCODE would yield if ran. STARCODE is deterministic, i.e., inputing the same parameters will always yield the same output spectrum.
I have a considerable number of pre-computed STARCODE models that I can use to train this neural network. I had originally split these precomputed models into training and validation samples and trained the neural network in a standard way, checking the mean squared error of the validation set to ensure that I do not overtrain. However, when I looked at the MSE of the validation set as a function of training iteration, I found out that it just gets at a minimum value and stays around it (see figure), instead of increasing again as one would expect once overtraining occurs.
The resulting neural network does a great job predicting the output spectra of STARCODE, i.e., when given the same input parameters, the real STARCODE output and the prediction from the neural network are very similar (enough for my purposes). Using a third set of STARCODE spectra that were not used for training or validating, there is no sign of overtraining: the artificial neural network predicts them correctly too. I want to confirm a couple of ideas before I start using the neural network for the analysis:
- I think the MLE behavior arises from the fact that these models do not have any noise in them. Is this correct? If so, does it mean that it is not possible to overtrain the neural network in this particular case?
If that is also correct, I guess the most optimal way to train the neural network would be to:
1) determine the number of neurons by increasing it until the final MSE value does not improve anymore (something that I am doing already), and then
2) once I have estimated the optimal number of neurons, I do not need to use a validation set since, if I am right, the neural network cannot be overtrained.
Is my interpretation of the MSE behavior correct, or am I missing something?