1
$\begingroup$

I am using artificial neural networks for an unconventional problem and, although it looks like it is working, I want to make sure that I am not doing anything wrong.

I have a code, STARCODE, that predicts the spectrum of stars based on some their physical properties (input parameters). STARCODE solves many physical equations and it takes several hours to run it. At the end of this process, it yields the corresponding spectrum of that star. I want to use STARCODE to fit a large sample of observed spectra, but the long time required to compute each model prevents me from using statistical methods such as MCMC. My idea is to train a neural network that receives the same input as STARCODE, and outputs a spectrum similar to what STARCODE would yield if ran. STARCODE is deterministic, i.e., inputing the same parameters will always yield the same output spectrum.

I have a considerable number of pre-computed STARCODE models that I can use to train this neural network. I had originally split these precomputed models into training and validation samples and trained the neural network in a standard way, checking the mean squared error of the validation set to ensure that I do not overtrain. However, when I looked at the MSE of the validation set as a function of training iteration, I found out that it just gets at a minimum value and stays around it (see figure), instead of increasing again as one would expect once overtraining occurs.

Mean Squared Error

The resulting neural network does a great job predicting the output spectra of STARCODE, i.e., when given the same input parameters, the real STARCODE output and the prediction from the neural network are very similar (enough for my purposes). Using a third set of STARCODE spectra that were not used for training or validating, there is no sign of overtraining: the artificial neural network predicts them correctly too. I want to confirm a couple of ideas before I start using the neural network for the analysis:

  • I think the MLE behavior arises from the fact that these models do not have any noise in them. Is this correct? If so, does it mean that it is not possible to overtrain the neural network in this particular case?
  • If that is also correct, I guess the most optimal way to train the neural network would be to:

    1) determine the number of neurons by increasing it until the final MSE value does not improve anymore (something that I am doing already), and then

    2) once I have estimated the optimal number of neurons, I do not need to use a validation set since, if I am right, the neural network cannot be overtrained.

Is my interpretation of the MSE behavior correct, or am I missing something?

$\endgroup$

bumped to the homepage by Community 2 days ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

  • $\begingroup$ What do you mean that there is no noise? If the data is deterministic, then you don't need machine learning for it... $\endgroup$ – Tim Sep 30 '17 at 18:43
  • $\begingroup$ The models (the code that I had in the first place) are deterministic, i.e., the same parameters will always yield the same result. However, evaluating them with the code that solves the physical equations takes hours, and it is unfeasible to use it for statistical analysis. I am using machine learning to predict the output of the models based on the input parameters, but without having to wait for hours for each evaluation. Is there a better approach for this? $\endgroup$ – Álvaro Sep 30 '17 at 18:47
  • $\begingroup$ So you want to find approximate solution, and then there will be noise since the approximate solution will not fit data perfectly. I see no reason to assume that the model can't overfit: it can, it simply can learn some rules that let it make perfect predictions, but do not make sense. $\endgroup$ – Tim Sep 30 '17 at 18:51
  • $\begingroup$ I have edited the question with (I think) a better description of the problem. I am concerned about overtraining the ANN (overfitting observations with it later on is of course possible, but that I will take care when I reach that stage). If overtraining is possible in this situation, then what could be the reason for the MSE not increasing after a certain iteration? It that pointing to some problem in the way I have done the training/validation process? $\endgroup$ – Álvaro Sep 30 '17 at 19:30
0
$\begingroup$

Your MSE plot is really supposed to also contain a line for the validation MSE and testing MSE. The MSE of the validation may go up -- as you eluded to. For continuous input features, sometimes when using an ANN for function approximation, e.g. fitting a 3D surface curve, the spacing of the observed data can cause the ANN to literally bomb out (totally fail), so the workaround is to employ Latin Hypercube Sampling (LHS) when the input is essentially a multi-dimensional grid. You mentioned input spectra, so it sounds like your input is maybe an energy-steradian grid of some sort. (so it sounds like you are not using the ANN to classify discrete objects, but rather predict using all continuous data).

Other than the above issues, other things you might consider is use of "cross-entropy" instead of MSE -- so see Bishop's classic ANN book available free online. Since many of your input spectra result in a matrix essentially being fed to the ANN, try using PCA to reduce dimensions and accept dimensions whose eigenvalues $\lambda_j>1$ as a start. That way you will prevent correlated data from being used in the ANN. Correlated data slow down an ANN's learning, since it wastes time learning the correlation. Orthogonality is the best thing to feed an ANN. FYI - PCA is also a very common technique to denoise data for input into an ANN.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.