I have created a Neural network that gets its training data from a complicated physics simulation. I run the simulation by randomizing 7 different inputs. Each input can be 1 of 4 discrete values. I have decided to use an input vector with 28 binary nodes (7 inputs * 4 possibilities per input). The output layer is just 1 node, a continuous number. This is a controlled simulation so there must exist a true relationship between these inputs and my output, right?
No matter what architecture or method I use, the accuracy seems to hit a wall at 20%. Without going into the specifics of my cost/accuracy function, that is about the number I expect if the neural net just outputs a random number in the range of all my target data (not good). Now the neural net can perfectly fit the training data if I make it very deep. But bad results notwithstanding, I get the lowest TEST score reliably by making the network as shallow as possible, i.e. a single hidden layer with 1 or 2 nodes. Clearly things are amiss.
- If I give the hidden layer ~20 nodes the test score starts moderately low (accuracy ~20%) and rises as overtraining kicks in. The output looks something like this. No matter what this network accuracy hits a wall around that 20% area.
- Increasing the training set seems to have diminishing returns
Please help, I am eager to see if anyone else has had behavior like this. I will readily provide any other information you need to better understand the problem.
Many thanks in advance. -Joel
Addendum: If I change the inputs to 7 (different) continuous variables then the relationship can be captured reasonably well with many hidden nodes. If I mix in the discrete variables with those continuous variables then the generalization falls apart again. This is an elusive problem, but if I had to guess it seems that the big issue is using these discrete variables, represented a stacked one-hot vectors. It would be really nice to find their relationship to the output because they are important inputs to the simulation. Perhaps it would help to see examples of Neural Networks that have worked using similar, discrete inputs represented as one-hot vectors or otherwise.