Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm training a 4 layer (2 hidden layers) Neural Net, with 10 input nodes, and 6000 output nodes.
During training, my loss function (RMS), and R2 currently looks like this:
So far it seems like the Loss function is starting to level off, even though my R2 is only at 0.2.
Is it reasonable to expect that there will be another sharp improvement? Or do Neural Networks typically go through a single "phase" of improvement?
Judging by the way the charts look, I would say there is room for improvement for R^2. You can probably wait for more iterations until the loss function reaches a flat zone.
You can also experiment starting with a higher learning rate and apply a decay during training.
Or do Neural Networks typically go through a single "phase" of improvement?
I think this depends more on the data you are using for training. There can be cases where training might get stuck in a local optimum making the network for example to predict correctly only 3/5 classes and just randomly guess for the other 2. If the limit between classes is sensible, the training might exit the local optimum and you will see another sharp descend in the error function.
