I have a convolutional network (For details, please see the edit in the bottom) with training/testing errors that always look very similar to what is shown in the figure. In other terms, it seems that my model is overfitting without any generalization whatsoever from the begining. What I find also bizarre is its oscilliation, which is not present in the training error. I've tried enlarging the dataset, but the effects were almost negligible (the test error decreases for, say, 10 iterations, and then it starts oscilliating again.).

Am I wrong to think that a model that can overfit the training data is capable of modeling the problem? How should I interpret these errors?

enter image description here

Edit: The netwok architecture is given in the following figure. Basically, it takes two images as input. Net1 and Net2 are both simple convolutional architectures made of Convolutions, MaxPooling, and PRelu. I haven't used dropout, but I use batch normalization. The term SPP stands for spatial pyramidal pooling.

As for the dataset,I just randomly select my training/testing dataset (90% of all images as training data, 10% as testing). But this doesn't seem to affect the shape of the errors. enter image description here

  • 1
    $\begingroup$ Welcome to CV. In order to increase the likelihood of an answer, please add some details: what is your application? What are the features and what is the response? How did you split between training and test set? Can you describe in detail your network architecture, including the kind of activation functions, if you're using dropout or not, etc.? $\endgroup$
    – DeltaIV
    Jul 9 '17 at 11:45
  • 1
    $\begingroup$ Without any detail, I could only suspect that the joint distribution of data is different between training and test set, but there could be literally dozens of other causes. $\endgroup$
    – DeltaIV
    Jul 9 '17 at 11:48
  • $\begingroup$ @DeltaIV: could you please give me a few pointers to some of the dozens of other causes? Even if it doesn't solve the problem I'd like to know about them. Thanks. $\endgroup$
    – Ash
    Jul 9 '17 at 12:39
  • $\begingroup$ We still don't know your application (why two images? What does the NN have to do with them? Classify them? Count the number of cats? :) ) and the actual architecture. See stackoverflow.com/questions/40147568/… for an example of what I meant. Incidentally, it shows one of the other possible causes (bad initialization, though with PReLU and batch normalization I wouldn't have expected bad initialization to be a big issue...but you never know). $\endgroup$
    – DeltaIV
    Jul 10 '17 at 5:12
  • $\begingroup$ I'm wondering if this would be more fit for Stack Overflow...I'm not sure. $\endgroup$
    – DeltaIV
    Jul 10 '17 at 5:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.