1
$\begingroup$

I want to train a neural network to classify a few simple, cartoony images like the ones below (for the moment I only have the classes house, tree, and sword).

house tree sword

The images I am (currently) using are downsized to 32x32 pixels, and the feed-forward network architecture I use is 1024-512-256-3. This means that I end up with a total number of weights (excluding biases) of

1024*512 + 512*256 + 256*3 = 656128

That is a huge number and some function optimization algorithms end up depleting the available memory because of it.

Obviously I'm doing something wrong. Should I use a different architecture or a different type of neural network (not MPL)? Should I reduce the image sizes further? When people train neural nets for complex object recognition tasks, how to they avoid using up all the memory?

$\endgroup$
1
$\begingroup$

You have a huge number of weights and need to regularize your network. Convolutional networks are a good option: you can have a deep network and at the same time be able to reduce the number of necessary parameters. Yann LeCunn (the guy who introduced them) has a bunch of nice papers describing it.

This technique refers to the more general concept of "weight sharing" and soft "weight sharing". The idea is to enforce group of weights to have the same (weight sharing) or similar values (soft weight sharing). The reason behind it is that you preencode some invariances in your architecture because you know that your problem present them (for example, you assume that the object identity is independent of where the object is located).

Another option, as you already point out, is try some optimization method which attempts to regularize your weights (weight decay, dropout, ...)

$\endgroup$
  • $\begingroup$ The problem is not that the network overfits the data. I'm not able to train it because I run out of memory. I tried using Scipy's fmin_bfgs to find the weights for which the output error is minimal. The problem is that the large number of weights (656.128) causes the algorithm to run out of memory. It tries to create an identity matrix eye(656128), with a total of 430.503.952.384 elements. $\endgroup$ – Paul Manta Mar 7 '13 at 18:27
1
$\begingroup$

This number of features is often seen with MLP architectures.

some function optimization algorithms

I'm afraid you're not using the right learning algorithms for training the weights of your network. Less than 1 million features should not create memory problem with iterative layer-wise methods, like the Backpropagation algorithm (but it usually takes time on a simple computer). For more efficient methods, take a look at the recent literature on deep learning algorithm (Bengio, LeCun & others).

$\endgroup$
  • $\begingroup$ Doesn't "backpropagation" refer to simply calculating the error gradient wrt. to the weights? The actual optimization algorithm I tried using is BFGS. $\endgroup$ – Paul Manta Mar 7 '13 at 18:22
  • 1
    $\begingroup$ BFGS usually has quadratic space complexity (O(n^2) => e.g. 600k**2*4 > 1 GB). Maybe L-BFGS would work better. For big neural networks most people use variants of stochastic gradient descent (SGD). $\endgroup$ – alfa Mar 7 '13 at 19:23
  • $\begingroup$ True. The SGD with small mini-batches (~16,32,64) is really simple to compute and often gives very good results. $\endgroup$ – Emile Mar 8 '13 at 8:10
  • $\begingroup$ @alfa Thanks for pointing that out. I'm going to try L-BFGS, since that's implemented by the library I use (Scipy). Do you know where I might find space and time complexity specs of other algorithms? $\endgroup$ – Paul Manta Mar 8 '13 at 8:36
  • 1
    $\begingroup$ @PaulManta The book chapter Efficient Backpropagation from Yann LeCun is a good starting point for neural networks optimization algorithms. It is freely available from his website. $\endgroup$ – alfa Mar 8 '13 at 10:08
1
$\begingroup$

There are some substantial preprocessing steps that I am not seeing here. You can transform the information in ways that are radically different than taking a geographic average as you do when downsampling, and that throw away much less information.

Options:

  1. Consider making an empirical CDF of the intensity values in the
    images and using them as inputs. If you have 8-bit pixels then this is only 255 inputs.
  2. Consider making an empirical CDF of the intensity values in sub-panes of the image (8x8 panes) and using them as inputs.
  3. Consider using GMM on your image to approximate some level of rescaled intensity values for your images, then reversing them such that when the image intensity values are input the probability of membership is output. Use the membership probability as input to your model.
  4. Perform a Gaussian smooth on numeric derivative over the image for "approximate edge" location (or use Sobel or whatever) and then perform the above transformations and use as inputs to your network.

I think that idea 1 is your best bet for good classification on very sparse data. You are engaging the "curse of dimensionality" and if you don't have lots of compute resources and lots of training data then you can get results that are pretty meaningless.

$\endgroup$
0
$\begingroup$

How did you choose this particular network architecture? Why don't removing a layer or reducing hidden neurons?

Maybe you can also try to reduce the problem dimensionality using PCA (or something similar) and/or trying with unsupervised methods (clustering methods, SOM). You can find a lot of good material about Computer Vision on Internet.

$\endgroup$
  • $\begingroup$ No particular reason I chose this architecture. I wanted to try multiple architectures, to see which one was best in terms of performance-size ratio. $\endgroup$ – Paul Manta Mar 7 '13 at 18:41
  • 1
    $\begingroup$ Anyway, try to remove a layer and/or avoid training algorithms that use second derivatives, because usually they take a lot of memory (e.g. MATLAB trainlm). $\endgroup$ – Matteo De Felice Mar 8 '13 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.