I recently came across a fascinating discussion on how a simple logistic regression model can achieve around 92% classification accuracy on the MNIST dataset (reference: How does a simple logistic regression model achieve a 92% classification accuracy on MNIST?).
For a linear model, 784 neurons (one for each pixel) seem to be the minimal configuration. However, I’ve been exploring how far we can push minimalistic models further while still achieving high performance. I’ve developed a neural network with only 702 parameters, and it achieves 98.2% accuracy on MNIST. You can find the implementation of my model here: 702-parameter MNIST model.
I’m curious if there are other minimalist approaches—like neural networks with very few parameters—that can achieve accuracy beyond 92%, possibly approaching 98-99%, without resorting to highly complex architectures.
Specifically, I’m looking for:
- Examples of minimalist neural networks or other models that achieve high accuracy on MNIST with fewer than 1,000 parameters.
- Techniques or tricks (e.g., special regularization, weight initialization, or activation functions) that help such models maintain high performance while remaining computationally efficient.
Any insights or references to papers that explore this minimalist yet powerful modeling direction would be greatly appreciated!
