4
$\begingroup$

I created forecasts using ARIMA, Multilayer Perceptron (MLP) and Extreme Learning Machine (ELM) and would like to compare their accuracies. It's straightforward with the ARIMA. However, the outcomes of MLPs and ELMs change every time I run them. So, how do I deal with those changing outcomes when measuring accuracy?

Should I just use set.seed() (I am using R) and go with one specific outcome or use a mean of, say, ten forecasts?

Moreover, I often read about forecasts with "accuracy of X %" and I am at a loss at how to calculate such absolute accuracy for forecasts. At the moment, the measures I use are MAPE, MASE, RMSE, MAE.

$\endgroup$
3
$\begingroup$

Yes it is a very good practice to set a seed when dealing with random algorithms. If you see that the accuracy measurement is very fluctuant, you can run it several times (like ten, if you can afford the computation time) and average the result.

Keep in mind that you must compare accuracies of models based on a test set (not used in training) or in a cross-validation scheme.

The MAPE error gives you an accuracy in %, so if your MAPE is 10, you have an error of 10% in average.

$\endgroup$
2
$\begingroup$

I agree with @jlesuffleur that a set of algorithms with randomization should be used. Be aware though, that ideally a MLP should converge to a set of weights and random initialization is only supposed to accelerate getting there faster.

If the results of your MLP vary a lot between different weight initialization that could be a hint that it does not learn a lot about the data at all. Make sure that the number of training samples is larger than the number of independent parameters (number of weights), which can go into the thousands for MLPs. Otherwise, there is little hope of getting meaningful predictions from the MLP.

On the topic of error measures, the mean average percentage error is readily understood by laymen and therefore quite suitable when talking to customers etc. For model selection, you should use a different measure, such as the mean absolute error or root of mean squared error.

The reason is that the MAPE is more susceptible to errors in locations where the original value is small. A deviaton of 10 units contributes 10% when the true value was 100, but only 2% when the true value was 500.

The MAE and RMSE give you values that are hard to interpret, but better suited for comparing differnt models. The RMSE punishes large deviations more than the MAE, but which of the two to emphasize is up to you and the project's requirements.

$\endgroup$
  • $\begingroup$ You write that the number of parameters should be smaller than the number of observations. This is only true in the case of networks that are not subject to regularization, either via L1/L2 weight decay, or dropout. $\endgroup$ – generic_user Nov 28 '17 at 14:43
  • $\begingroup$ Thank you for the long answer! I'll take it into account when evaluating my models $\endgroup$ – Clemens Kaiser Nov 28 '17 at 16:40
  • $\begingroup$ @generic_user I am not sure I can follow you to that conclusion. What I'm alluding to is that if the mapping of model parameters to outputs has full rank you need exactly N conditions to fix N parameters uniquely. Ideally, you want more conditions than parameters to enable your model to generalize properly. I don't see how dropout or regularization change anything in this context. $\endgroup$ – Neuneck Nov 29 '17 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.