2
$\begingroup$

I use a MLP with one hidden layer (15 nodes) and one output node. I use a sigmoid activation function, atan as error function, the error itself is calculated with MSE, 5-fold crossvalidation, resilient backpropagation for a batched binary classification task where within each batch approx. 1000 samples are available.

My original dataset has a ratio of approx. 30/70 positive vs. negative samples. No matter what NN setup I tried (more features, more samples) the training error didn't go beneath 0.1, the f-measure I used for evaluation was between 0.3-05, precision 0.6-0.8 and recall only between 0.2-0.4.

Then I tried oversampling in order to increase the positive/negative value to approx. 1. Now with the same setup I the error decreased only to 0.09, but now I get a constant f-measure of > 0.85 , precision around 0.8 and recall 0.95-1(!?).

Now I'm really wondering if my setup is completely wrong or if have found a way to fit my data well.

Does anybody have some hints where I might have made a mistake or do you think my setup is ok and my classifier, too?

Improve this question
$\endgroup$
2
  • $\begingroup$ Have you tried adding skip-layer connections? I usually run skip connections with a weight decay (aka ridge penalty). If all the hidden -> output connections get shrunk to zero, then I tend to believe the interactions are weak sources of information for a given data set... $\endgroup$
    – Shea Parkes
    Feb 2 '12 at 21:26
  • $\begingroup$ I removed the spoiling feature. Without oversampling now my error is constantly at 0.2 and the f-measure around 0.2-0.4. With oversampling the error stays almost the same, but the f-measure goes up to 0.7-0.8. So now back to my original question: Is my setup and the oversampling approach completely wrong? $\endgroup$
    – Andreas
    Feb 6 '12 at 6:37
2
$\begingroup$

I found a spoiling feature which caused a linear dependency between input and output. That's why my upsampling increased the measurement values so good.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.