What are the real hyperparameters of a neural network? What are they? Learning rate, epochs, mini-batch size, and what else?
Do I consider network architecture as hyperparameter too?
I didn't found a concrete answer for the last question.
 A: Yes. Essentially, any parameter that you can initialize (before training the neural network model) can be seen as a hyperparameter.
This includes the optimizer's hyperparameters (e.g., SGD, Adam, etc.): learning rate, decay rates, step size, and batch-size; as well as model's hyperparameter (CNN): number of layers, number of units at each layer, drop out rate at each layer, L2 (or L1) regularization parameters, activation function type (ReLU, Sigmoid, Tanh), and if you are dealing with CNNs, there are extra hyperparameters such as the ones related to convolutional layer: window size, stride value, and Pooling layers.
There are even more hyperparameters that you can initialize and tune. For example, take a look at this list.
A: Hyperparameters for a deep neural network:
- Number of iterations
- Number of layers LL in the neural network
- Number of hidden units in each layer
- Learning rate α
- Step size
- Choice of the activation function
- Losss function
- Mini-batch Size
- Momentum
- Regularization
- Drop out rate
- Weight Decay
