2
$\begingroup$

I have a binary classification problem for financial ratios and variables. When I use newff (with trainlm and mse and threshold of 0.5 for output) I have a high classification accuracy (5-fold cross validation – near 89-92%) but when I use patternnet (trainscg with crossentropy) my accuracy is 10% lower than newff. (I normalized data before insert it to network - mapminmax or mapstd)

When I use these models for out-sample data (for current year- created models designed based one previous year(s) data sets) I have better classification accuracies in patternnet with better sensitivity and specificity. For example I have these results in my problem:

Newff:

Accuracy: 92.8% sensitivity: 94.08% specificity: 91.62%

Out sample results: accuracy: 60% sensitivity: 48% and specificity: 65.57%

Patternnet:

Accuracy: 73.31% sensitivity: 69.85% specificity: 76.77%

Out sample results: accuracy: 70% sensitivity: 62.79% and specificity: 73.77%

Why we have these differences between newff and patternent. Which model should I use?

Thanks.

Improve this question
$\endgroup$
1
  • $\begingroup$ Please avoid posting the same question on different StacKExchange forums ( stackoverflow.com/questions/23649621/… ); if the question is deemed out-of-context it will be transfered by our moderators. $\endgroup$
    – usεr11852
    Commented May 16, 2014 at 21:52

1 Answer 1

Reset to default
4
$\begingroup$

On face value I would recommend using patternnet as it gives you better out of sample performance; the results from newff seems suspiciously good leading me to believe some over-fitting occurs. On that matter check the following link: Improve Neural Network Generalization and Avoid Overfitting.

To comment on the different results: For newff a Levenberg-Marquardt backpropagation is utilized while for patternnet, scaled conjugate gradient backpropagation. In general, different optimization procedures are not guaranteed to arrive in the same result even if they had the target function to optimize against. In your case through you are also using different target functions (mse and crossentropy respectively). It would probably be alarming you if did got the same results as you are fitting different criteria. :)

Having said that, using newff seems a bit odd. It is considered obsolete since R2010b and you are recommend (by the docs) to use feedforwardnet. Try using feeforwardnet first and then decide on which procedure you will ultimately use. As it stands it seems like you comparing the performance of a function (newff) people have not worked on for at least 4 years (if not more) against the performance of a function (patternnet) that is actively developed. It is not really surprising that the latter one it does a better job.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for answer. I added another question in above main question. I checked with Patternnet but have the same problem. (I think there isn’t any difference between structures of newff and patternent in designing network but patternnet is newer function so I must use it). I’m using early stopping in every structure, so besides using early stopping why I have over-fitting? My system structure mentioned above. thanks. $\endgroup$
    – user2991243
    Commented May 17, 2014 at 10:21
  • $\begingroup$ As mentioned because different fitting criteria are used, the same network structure will almost certainly be optimized for different tasks. And it is not necessary you get a 95%+ classification success in any case for your out-of-sample predictions. Finally, it might also be the case that your training sample is not a good approximation of your population. So while doing a good initial training, the network has not learned "enough" true populations characteristics. (Ah! And you can always experiment with different transfer functions; currently you are using tansig in both cases I guess.) $\endgroup$
    – usεr11852
    Commented May 17, 2014 at 11:25
  • $\begingroup$ Thank you so much for your help. what's your idea about my second question (PS. in main question that added today)? optimization algorithm etc. $\endgroup$
    – user2991243
    Commented May 17, 2014 at 11:43
  • $\begingroup$ Bayesian regularization in NN's is far from my area of expertise, I am sorry. I think it will probably worth your time starting a new question about it. It does have a significant statistical context so I believe you will be able to get some good answers. As a general suggestion: Try not to pack a lot of different questions in one thread because it is harder to draw people's attention to it as well as makes it harder for someone in future to guess what the context of a thread is. $\endgroup$
    – usεr11852
    Commented May 17, 2014 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.