0
$\begingroup$

First, I am a computer scientist and only know the very basic of statistics.

I am visualizing the learning behaviour of different machine learning architectures (VGG, ResNet, etc.) by using the CIFAR-10 dataset. I do that by training the models for multiple epochs and print the misclassification rate after every 5th epoch in a line plot.

For example, the misclassification rate of cat images which were falsely predicted as dog images over 195 epochs for a VGG and a ResNet model looks like this:enter image description here

The blue line is VGG and the orange is ResNet. As you can see for example, the confusion rate for ResNet is often higher (e.g. epoch 58 and epoch 120) compared to the VGG model as ResNet mispredicts more cat images as dog images.

Another example is the precision of both models: enter image description here

The ResNet seems to start (epoch 0) with a lower misclassification rate compared to VGG.

So my actual question: I want to analyze these learning statistics more deeply. I want to find out if there are any patterns or characteristics in the line graphs. I have been reading about "time series analysis" in the last couple of days. I think this could be a good idea but I am still not sure if it made sense to apply models like autoregressive models, ARMA, ARIMA to this kind of data. Can you give me advice please?

$\endgroup$
2
  • $\begingroup$ You could also use an RNN, and use a Deep neural network to study the learning statistics of Deep Neural Networks. So meta 😀 $\endgroup$
    – DeltaIV
    Apr 18, 2018 at 13:02
  • 1
    $\begingroup$ Google TensorFlow folks like to tell anecdotes about their AI learning to optimize and control heating and cooling of the building where it's housed $\endgroup$
    – Aksakal
    Apr 18, 2018 at 13:36

1 Answer 1

0
$\begingroup$

If you want to observe patterns in these models, use visualization or model selection techniques. I don't understand why you would use time series analysis to compare these models. Let me know if I'm missing something about time series.

$\endgroup$
1
  • $\begingroup$ You are right. Maybe I didn't state that clearly. For comparing, I use visualization techniques. That already gives me some insight. But I think I can get more out of the line graphs by analysing each line by its own. So I don't want to compare the models now. I just want to find patterns or characteristics for each line. $\endgroup$
    – NMO
    Apr 18, 2018 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.