2
$\begingroup$

I beginning to learn about neural networks and how to train them. I've been reading about using gradient descent for training. Most books go straight to the backpropogation algorithm for computing $\nabla C$, where $C$ is the cost function.

I wanted to try computing $\nabla C$ by using finite differences: $$\frac{\partial C}{\partial w_i}=\frac{1}{h}\left[C({w_i+h})-C({w_i})\right]$$ where $w_i$ is some weight in the network. (Please forgive my sloppy notation.)

I've coded this algorithm, but when I try to train my network on the MNIST data, calculating the gradients takes forever. Obviously, this is why everyone recommends backpropogation.

Nonetheless, I'd like to see if my finite difference gradient descent training algorithm works. I'd like to compare against Scikit-Learns solution to a simple problem.

Can anybody recommend a small problem that is amenable to solution with a simple neural network? Preferrably, I would be able to train on a small amount of data with a relatively small feature space.

$\endgroup$
2
  • 1
    $\begingroup$ I agree with @Sycorax. MNIST is about as small as they get, unless you want to create your own toy data set. Or perhaps use a REALLY simple data set where you're predicting height by weight and age or something like that. $\endgroup$ Oct 22 '20 at 22:08
  • 1
    $\begingroup$ @Sycorax I think I will start with the XOR problem and see how it goes! $\endgroup$ Oct 22 '20 at 22:09
1
$\begingroup$

The easiest path forward is to just reduce the size of the MNIST data by 99% or more by discarding most of the examples. You could even do something as extreme as only keeping 20 images, 1 of each class as training data and another of each class as test data. This is the smallest you could possibly make the MNIST data without doing something fancy like distillation.

Another simple solution is to train a model on the XOR problem. The Tensorflow Playground has a number of in-browser toy problems to play with. https://playground.tensorflow.org/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.