I use a MLP with one hidden layer (15 nodes) and one output node. I use a sigmoid activation function, atan as error function, the error itself is calculated with MSE, 5-fold crossvalidation, resilient backpropagation for a batched binary classification task where within each batch approx. 1000 samples are available.
My original dataset has a ratio of approx. 30/70 positive vs. negative samples. No matter what NN setup I tried (more features, more samples) the training error didn't go beneath 0.1, the f-measure I used for evaluation was between 0.3-05, precision 0.6-0.8 and recall only between 0.2-0.4.
Then I tried oversampling in order to increase the positive/negative value to approx. 1. Now with the same setup I the error decreased only to 0.09, but now I get a constant f-measure of > 0.85 , precision around 0.8 and recall 0.95-1(!?).
Now I'm really wondering if my setup is completely wrong or if have found a way to fit my data well.
Does anybody have some hints where I might have made a mistake or do you think my setup is ok and my classifier, too?