2
$\begingroup$

I have created a Neural network that gets its training data from a complicated physics simulation. I run the simulation by randomizing 7 different inputs. Each input can be 1 of 4 discrete values. I have decided to use an input vector with 28 binary nodes (7 inputs * 4 possibilities per input). The output layer is just 1 node, a continuous number. This is a controlled simulation so there must exist a true relationship between these inputs and my output, right?

No matter what architecture or method I use, the accuracy seems to hit a wall at 20%. Without going into the specifics of my cost/accuracy function, that is about the number I expect if the neural net just outputs a random number in the range of all my target data (not good). Now the neural net can perfectly fit the training data if I make it very deep. But bad results notwithstanding, I get the lowest TEST score reliably by making the network as shallow as possible, i.e. a single hidden layer with 1 or 2 nodes. Clearly things are amiss.

  • If I give the hidden layer ~20 nodes the test score starts moderately low (accuracy ~20%) and rises as overtraining kicks in. The output looks something like this. No matter what this network accuracy hits a wall around that 20% area. enter image description here

And 2 nodes for reference enter image description here

  • Increasing the training set seems to have diminishing returns

Please help, I am eager to see if anyone else has had behavior like this. I will readily provide any other information you need to better understand the problem.

Many thanks in advance. -Joel

Addendum: If I change the inputs to 7 (different) continuous variables then the relationship can be captured reasonably well with many hidden nodes. If I mix in the discrete variables with those continuous variables then the generalization falls apart again. This is an elusive problem, but if I had to guess it seems that the big issue is using these discrete variables, represented a stacked one-hot vectors. It would be really nice to find their relationship to the output because they are important inputs to the simulation. Perhaps it would help to see examples of Neural Networks that have worked using similar, discrete inputs represented as one-hot vectors or otherwise.

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ General rule: total number of free parameters in your model should be less than 10% of the number of training points. Hopefully a lot less, like 1% or 0.1%. To do this, make the model shallower and/or narrower until your training error is about equal to your test error. Then try to look deeper into the problem and develop a model with the fewest number of free parameters. It is likely that your best model won’t involve neural networks at all, but physical principles instead! Hurray! $\endgroup$
    – Peter Leopold
    Commented Feb 15, 2019 at 13:50
  • $\begingroup$ Hey Peter: Even if the relationship can be boiled down to phsyical principles right now I am trying to do a proof of concept. The Neural Network is clearly not capturing the relationship at the moment. Even if a Neural network is overkill, it still SHOULD be able to model the behavior of the relationship, right? $\endgroup$
    – jlapin
    Commented Feb 15, 2019 at 14:51
  • 1
    $\begingroup$ You can overtrain any learning model to memorize the training set, but memorization is the opposite of learning. You can memorize uncorrelated Gaussian noise, for instance, but if you try to learn it, a successful model should give the only mean and have a steady L2 loss equal to the variance. The fundamental goal of learning is for loss-in-training to be approx equal to loss-in-testing. Yes, a well designed NN should achieve that goal, even and especially if your data is pure noise. Try developing a model from trivially simple up. E(train) ~< E(test) at the start. Try to keep it that way. $\endgroup$
    – Peter Leopold
    Commented Feb 15, 2019 at 15:15
  • $\begingroup$ That gets to the heart of my problem. With the shallowest network possible (1-2 hidden nodes) I can get my training and test costs to flatten out at around 10 epochs, whereas with the deeper network the training cost continued to drop while the test cost rose. But my accuracy keeps hitting the 20% ceiling, which as I said is probably not much higher success than guessing the output at random. Have you ever seen such behavior, i.e. a floor/ceiling for the test cost/accuracy, largely irrespective of choice for the hyperparameters? I just added another picture for 2 hidden nodes. $\endgroup$
    – jlapin
    Commented Feb 15, 2019 at 18:26
  • $\begingroup$ Hello, can you please describe what your output is? is it a classification problem? From what you described, it seems one of the 2 following things is happening: your simulation is wrong and the input does not carry any information about the output. Or it does but you are overfitting right away. I would first test if it is indeed number 1 by trying an algorithm with less complexity such a linear/logistic regression or trees/svm for instance. If you get better results, then you may want to look again at your NN. $\endgroup$
    – Tom
    Commented Feb 15, 2019 at 19:20

1 Answer 1

Reset to default
1
$\begingroup$

It might be that the problem is so hard that the training points you have are not enough to estimate the underlying decision function(e.g. in a checker-board pattern, you would only expect to obtain a decent estimate once you are covering a big part of the squares). Have you played around with the number of training points? How many points do you have available?

have you tried other approaches? Neural networks are probably the most fickle model to throw on an arbitrary problem. What does logistic regression say? How about a random forest? Both algorithms have some basic ways to analyze their decision function and measure how important certain variables are to the decision boundary.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hey Ulfgard: I just tried running it many times by dividing my 1000 samples into smaller and smaller sets, and unfortunately the relationship with test cost looked pretty flat. You might be right that the problem is too hard/relationship too weak. Like I said earlier, changing the discrete value refers to change of pulse phase in the simulation, which means an entirely different matrix is being applied in the calculation. But I worry this is a larger problem with representation of discrete inputs, and can think of one other example of a NN failing in the same way, possibly for the same reason. $\endgroup$
    – jlapin
    Commented Feb 16, 2019 at 12:37
  • $\begingroup$ discrete inputs are always challenging. any chance providing the network with the original, non-discretized values? $\endgroup$
    – Ulfgard
    Commented Feb 16, 2019 at 16:12
  • $\begingroup$ When you say "original, non-discretized values?", what are you referring to? Unfortunately I don't think there is any way of representing these attributes of the simulation as continuous values. $\endgroup$
    – jlapin
    Commented Feb 16, 2019 at 22:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.