# Small, simple neural network test problem?

I beginning to learn about neural networks and how to train them. I've been reading about using gradient descent for training. Most books go straight to the backpropogation algorithm for computing $$\nabla C$$, where $$C$$ is the cost function.

I wanted to try computing $$\nabla C$$ by using finite differences: $$\frac{\partial C}{\partial w_i}=\frac{1}{h}\left[C({w_i+h})-C({w_i})\right]$$ where $$w_i$$ is some weight in the network. (Please forgive my sloppy notation.)

I've coded this algorithm, but when I try to train my network on the MNIST data, calculating the gradients takes forever. Obviously, this is why everyone recommends backpropogation.

Nonetheless, I'd like to see if my finite difference gradient descent training algorithm works. I'd like to compare against Scikit-Learns solution to a simple problem.

Can anybody recommend a small problem that is amenable to solution with a simple neural network? Preferrably, I would be able to train on a small amount of data with a relatively small feature space.

• I agree with @Sycorax. MNIST is about as small as they get, unless you want to create your own toy data set. Or perhaps use a REALLY simple data set where you're predicting height by weight and age or something like that. Oct 22, 2020 at 22:08
• @Sycorax I think I will start with the XOR problem and see how it goes! Oct 22, 2020 at 22:09