0
$\begingroup$

Currently, I have made an ARIMA model and to validate it I have studied the autocorrelation (ACF and PACF) of residuals and also a Box-Pierce test.

But I am having doubts about how to validate a model made using ARTIFICIAL NEURAL NETWORKS.

Should I treat it as an ARIMA model and study residuals in the same way? Or should I make another consideration?

I was also wondering if the data used as validation has something to do with this (appart from telling me if there is or not overfitting).

All workship I find on the internet about it doesn't make any explicit validation (they don't assess the residuals or anything).

$\endgroup$
1
$\begingroup$

There are books on the topic. It is different because these are adaptive systems and so they can change themselves. While they could be stable currently, it does not mean they will not observe data that makes them unstable. See the following for examples:

Guidance for the Verification and Validation of Neural Networks by Laura L. Pullum, Brian J. Taylor, Marjorie A. Darrah

Independent Verification and Validation of Neural Networks - Developing Practitioner Assistance By Dr. Laura L. Pullum, Dr. Marjorie A. Darrah, and Mr. Brian J. Taylor, Institute for Scientific Research, Inc., Software Tech

Toward V&V of neural network based controllers by Johann Schumann and Stacy Nelson

Validating A Neural Network-based Online Adaptive System by Yan Liu, Dissertation submitted to the College of Engineering and Mineral Resources at West Virginia University, 2005

Verification and Validation of Adaptive and Intelligent Systems with Flight Test Results by John Burken and Dick Larson, UCAUV 2009

$\endgroup$
0
$\begingroup$

If you are creating a forecast of a continuous time series, you can do that.

This really depends on your objective though. If you are comparing your network model to the ARIMA, then you will probably want to use a similar metric so you can compare their accuracy. You're not treating it as an ARIMA model if you look at residuals. Residual analysis is widely used, as you probably know from GARCH models.

This said, have you ever heard of gradient boosting? It essentially defines an additive model where each model forecasts the residuals of the previous model's objective. To be clear, the first learner models the series, then the second learner models the first's residuals, and then the third models the residuals of the model of residual left by the second, etc. See article: https://en.wikipedia.org/wiki/Gradient_boosting and talk by Trevor Hastie from Stanford and H20.ai: https://www.youtube.com/watch?v=wPqtzj5VZus

It is commonly used for decision trees, but I am working with logistic boosting (something else) for neural networks in my project.

That was a slight digression though since the question was only about residual analysis. You should play around and see what you can come up with it. Maybe try clustering the residuals and see what happens. That's your choice and your exploration.

But you want to know about what you're dealing with. What are the limitations and assumptions in looking at residuals only? Are there any in this case? It's your data. You need to know it well.

Best of luck.

$\endgroup$
  • $\begingroup$ Tanks for answering, Sir. Actually, it is a time series, as you said in your first lines. I didn't know about "gradient boosting" but sounds really interesting. The problem is all the architectures I have trained of ANN's give a high R coefficient, but the resiudals present a high autocorrelation (as far as I know, this last means they are not random so I cannot accept the model). Which I find weird is that none of the 100 architectures I have tried passes that test. $\endgroup$ – Jvr Jun 14 '17 at 0:25
  • $\begingroup$ Are you using recurrent networks? If you're using feedforward networks than yea, it will be a pretty weak learner. Try a simple recurrent or LSTM. $\endgroup$ – Abraham Horowitz Jun 14 '17 at 11:41
  • $\begingroup$ In fact, I'm using LSTM. $\endgroup$ – Jvr Jun 14 '17 at 13:36
  • $\begingroup$ Sorry I haven't answered. This is pretty interesting. It could be that you aren't including enough time steps in your network, but otherwise, maybe this just isn't going to work with an RNN. Otherwise, maybe a neural network isn't the best way to go. Take a look at the models in pyhsmm package: github.com/mattjj/pyhsmm. What dataset are you using if you'd please? $\endgroup$ – Abraham Horowitz Jun 28 '17 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.