I'm working on a multi-class classification problem using a neural network, where my features are rather noisy, with some very similar inputs may belong to different classes for different training examples. On the other hand, I have a very large dataset, so this should offset that problem a bit.
I was wondering if there are general guidelines about good hyperparameters for neural networks for input features like this. There are a couple of things that seem logical to me, but I'm wondering if I'm correct in these thoughts, and if there are other assumptions I can make to move my hyperparameters in the right direction.
For instance, is it best to have a slowly decaying learning rate, as I don't want to "lock in" on observations too soon?
Another example: I guess I want to prevent neurons from dying too soon with data like this, how would I best accomplish this?
As indications for the correct class might be in small details, would I prefer multilayer networks vs a simpler network to better capture these details?