I have a binary classification problem for financial ratios and variables. When I use newff (with trainlm and mse and threshold of 0.5 for output) I have a high classification accuracy (5-fold cross validation – near 89-92%) but when I use patternnet (trainscg with crossentropy) my accuracy is 10% lower than newff. (I normalized data before insert it to network - mapminmax or mapstd)

When I use these models for out-sample data (for current year- created models designed based one previous year(s) data sets) I have better classification accuracies in patternnet with better sensitivity and specificity. For example I have these results in my problem:

#Newff:#

Accuracy: 92.8% sensitivity: 94.08% specificity: 91.62%

Out sample results: accuracy: 60% sensitivity: 48% and specificity: 65.57%

#Patternnet:

Accuracy: 73.31% sensitivity: 69.85% specificity: 76.77%

Out sample results: accuracy: 70% sensitivity: 62.79% and specificity: 73.77%

Why we have these differences between newff and patternent. Which model should I use?

Thanks.

PS.

I have an optimization algorithm that optimize numbers of neurons and early stopping (stopping after X). Besides it I use 5-fold cross validation in every iteration of optimization algorithm. As @usεr11852 said I have over fitting besides that I’m using early stopping? What is your idea about checking Bayesian regularization with “patternnet”? I have another problem in my system. The best output of optimization algorithm based on best classification accuracy results maximum early stopping that I had set for early stopping limitations. Why?