I am building a simple NN for a regression analysis (0.5 mln rows of data) in Keras.

# Define model
model = Sequential()
model.add(Dense(32, activation = 'relu'))
model.add(Dense(32, activation = 'sigmoid'))
# Define model parameters
model.compile(optimizer = 'sgd', loss = 'mse', metrics = [tf.keras.metrics.MeanAbsoluteError()])

I am satisfied with results, but would like to know if there is any rule of thumb or recommendations how to build such models for a regression analysis.

For example:

  • How many layers?
  • Which activation functions?
  • Which optimizer?

What is your state of the art approach, when you start to do such type of analysis?

Especially appreciate any academic sources.