1
$\begingroup$

I am relatively new to neural networks. To understand the basics, I converted the MNIST database into a format that I liked, and I wrote a single layer NN with 784 neurons from scratch without using any library related to NN.

I trained the NN over 600 samples, and I tested it on 10000 samples. (I accept that the reverse would be much better.) I can see that as the NN trains with more and more samples, the error decreases almost exponentially with the training sample size. However, in the end, the test error was 90%.

Is this normal for such a simple NN? Where can I find the performance of different types of NNs with different parameters to compare with my own NN?

Of course, this is a very simple NN, but before making it more complex, I would like to know whether the results I am getting are in agreement with others, and I would like to get an idea about what kind of different methods & schemes could be used in other situations.

Improve this question
$\endgroup$
4
  • $\begingroup$ Why aren’t you training on all 60000 training samples? $\endgroup$
    – Dave
    Commented Mar 14, 2020 at 11:29
  • $\begingroup$ @Dave current computational contraints. $\endgroup$
    – Our
    Commented Mar 14, 2020 at 11:30
  • 2
    $\begingroup$ You should set up the same model with an established framework and see what the performance is. 10% correct is very near to random guess rate. $\endgroup$
    – Jon Nordby
    Commented Mar 14, 2020 at 12:19
  • $\begingroup$ MNIST has 10 classes, so 90% error rate is no better than a random guess. Check for bugs. This problem is pretty easy, so any error rate higher than 10% suggests that whatever method you're using is deficient, has a bug, or both. $\endgroup$
    – Sycorax ♦
    Commented Oct 17, 2023 at 19:26

2 Answers 2

Reset to default
3
$\begingroup$

MNIST is a pretty simple, "toy" dataset, and you can get high performance ($>90\%$ accuracy) even with simple classifiers like logistic regression, single layer neural network, or $k$-NN. You can find summary of the results on the page maintained by Yann LeCun.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am not sure how this answers the question, given that the $90\%$ in the OP refers to the error rate and not the accuracy. Nonetheless, this has been accepted, so perhaps I am missing something. $\endgroup$
    – Dave
    Commented Oct 17, 2023 at 19:13
  • $\begingroup$ @Dave This appears to directly address the bold text in OP $\endgroup$
    – Sycorax ♦
    Commented Oct 18, 2023 at 1:53
2
$\begingroup$

THIS MAKES ME THINK THERE HAS BEEN A CODING ERROR

I remember this once happening to me. I had managed to scramble the true labels so that the expected accuracy really was just one out of ten for a $90\%$ error rate (because there are ten categories with even representation). Once I fixed that coding mistake I had made, my accuracy jumped up to be more along the lines of what would be anticipated for MNIST digit classification.

MNIST is pretty easy to solve with quite high accuracy, and while overfitting could happen, your model is so simple that I would suspect a coding error that has scrambled your test set categories before I would suspect a major issue with the model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The wrong style of training data or form of loss can do the same. Binary cross-entropy is not the same as categorical cross-entropy. $\endgroup$
    – EngrStudent
    Commented Oct 18, 2023 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.