I have a binary classification problem for financial ratios and variables. When I use newff (with trainlm and mse and threshold of 0.5 for output) I have a high classification accuracy (5-fold cross validation – near 89-92%) but when I use patternnet (trainscg with crossentropy) my accuracy is 10% lower than newff. (I normalized data before insert it to network - mapminmax or mapstd)

When I use these models for out-sample data (for current year- created models designed based one previous year(s) data sets) I have better classification accuracies in patternnet with better sensitivity and specificity. For example I have these results in my problem:

Newff:

Accuracy: 92.8% sensitivity: 94.08% specificity: 91.62%

Out sample results: accuracy: 60% sensitivity: 48% and specificity: 65.57%

Patternnet:

Accuracy: 73.31% sensitivity: 69.85% specificity: 76.77%

Out sample results: accuracy: 70% sensitivity: 62.79% and specificity: 73.77%

Why we have these differences between newff and patternent. Which model should I use?

Thanks.