Neural network hyperparameters for classification with noisy features

Question

I'm working on a multi-class classification problem using a neural network, where my features are rather noisy, with some very similar inputs may belong to different classes for different training examples. On the other hand, I have a very large dataset, so this should offset that problem a bit.

I was wondering if there are general guidelines about good hyperparameters for neural networks for input features like this. There are a couple of things that seem logical to me, but I'm wondering if I'm correct in these thoughts, and if there are other assumptions I can make to move my hyperparameters in the right direction.

For instance, is it best to have a slowly decaying learning rate, as I don't want to "lock in" on observations too soon?

Another example: I guess I want to prevent neurons from dying too soon with data like this, how would I best accomplish this?

As indications for the correct class might be in small details, would I prefer multilayer networks vs a simpler network to better capture these details?

Answer 1

The simplest answer to all of this is to grid search your network with a proper, nested cross-validation framework. Your parameter choices will be number of layers, nodes, learning rate (or other optimization parameters your algorithm may have), optimization function, regularization parameters, and activation functions.

It's a complex problem to try and pre-select these features. Certainly a more dense network will be better at approximating the underlying function describing your problem, but you still need to balance that vs. overfitting, which you can accomplish with proper regularization. You can also prevent vanishing gradients with a more complex activation function such as leakyReLU, but you then have another hyperparameter to tune. Exactly how deep, how regularized, how 'leakly' your activation function should be, are best explored with proper model building techniques, as each dataset will be unique.