Here's some straightforward practical advice for people who are struggling with the same questions. The answer by @Josh is quite elaborate but if you are like me, I wouldn't know how to implement it without some concrete examples. So here we go.
My assumption is that you mean multi-layer perceptrons (MLPs) by neural networks.
For all the following, I recommend to use the caret library as it provides a set of tools for benchmarking different models conveniently.
What number of hidden layers should I use when building the model?
You must optimize the network capacity, not the number of hidden layers alone. The network capacity is defined as the number of layers times the number of nodes per layer.
How does this choice affect the model?
There is a trade-off how this choice affects the model's performance: The higher the network capacity, the better it can predict the training data, but the more likely it will overfit them.
So you can narrow down by reducing a high network capacity step-by-step, or increasing from a low network capacity. E.g. if you start with one hidden layer, you can also use the
nnet library which is much faster than
Should I change the 'threshold' parameter from the default (0.01)?
If the computation takes too long or doesn't converge (RMSE values missing in output), then you should increase it. If waiting time is no problem for you (this is a subjective decision), then you can increase the
stepmax parameter from its default value (1e+05). Start with a mild increase, e.g.
stepmax = 1e+06 as the computation time will distinctly increase.
Will the choice of algorithm affect the prediction accuracy?
To answer this is definitely out of scope of this question, but the short answer is: Of course :)
Should I change any of the other parameters in neuralnet?
If your goal is to reach the best prediction performance, then it is by far faster to compare with other machine learning models. Random forests and gradient boosting machines are always good benchmarking candidates. With the caret library, you can use the same interface to train the models like this:
training_configuration <- trainControl(method = "repeatedcv", number = 10, repeats = 3)
method = "neuralnet",
data = training.set,
preProcess = c("center", "scale"),
threshold = 0.4, # narrow down from 0.5 to 0.1 in steps of 0.1
stepmax = 1e+06, # increase from default 1e+05 if increasing threshold doesn't help
trControl = training_configuration,
# for neuralnet, define the network layers
tuneGrid = data.frame(layer1 =3, layer2 = 3, layer3 = 3)
Should I include any other variables (lags, temporal variables etc) in my model?
Usually the more features, the better.
As you only have a few thousand sample observations, you must then do a little feature engineering:
Remove variables that have zero variance. These have no predictive value.
You can detect them simply by
Remove highly correlated features, as their predictive information is redundant.
You can detect that by a simple correlation plot e.g.
Also, would any other machine learning or statistical method be more suitable for this task?
This question is redundant with
Will the choice of algorithm affect the prediction accuracy?, see answer above.