Neural networks for simplistic image classification

Question

I want to train a neural network to classify a few simple, cartoony images like the ones below (for the moment I only have the classes house, tree, and sword).

The images I am (currently) using are downsized to 32x32 pixels, and the feed-forward network architecture I use is 1024-512-256-3. This means that I end up with a total number of weights (excluding biases) of

1024*512 + 512*256 + 256*3 = 656128

That is a huge number and some function optimization algorithms end up depleting the available memory because of it.

Obviously I'm doing something wrong. Should I use a different architecture or a different type of neural network (not MPL)? Should I reduce the image sizes further? When people train neural nets for complex object recognition tasks, how to they avoid using up all the memory?

Answer 1

You have a huge number of weights and need to regularize your network. Convolutional networks are a good option: you can have a deep network and at the same time be able to reduce the number of necessary parameters. Yann LeCunn (the guy who introduced them) has a bunch of nice papers describing it.

This technique refers to the more general concept of "weight sharing" and soft "weight sharing". The idea is to enforce group of weights to have the same (weight sharing) or similar values (soft weight sharing). The reason behind it is that you preencode some invariances in your architecture because you know that your problem present them (for example, you assume that the object identity is independent of where the object is located).

Another option, as you already point out, is try some optimization method which attempts to regularize your weights (weight decay, dropout, ...)

Answer 2

This number of features is often seen with MLP architectures.

some function optimization algorithms

I'm afraid you're not using the right learning algorithms for training the weights of your network. Less than 1 million features should not create memory problem with iterative layer-wise methods, like the Backpropagation algorithm (but it usually takes time on a simple computer). For more efficient methods, take a look at the recent literature on deep learning algorithm (Bengio, LeCun & others).

Answer 3

There are some substantial preprocessing steps that I am not seeing here. You can transform the information in ways that are radically different than taking a geographic average as you do when downsampling, and that throw away much less information.

Options:

1. Consider making an empirical CDF of the intensity values in the
   images and using them as inputs. If you have 8-bit pixels then this is only 255 inputs.
2. Consider making an empirical CDF of the intensity values in sub-panes of the image (8x8 panes) and using them as inputs.
3. Consider using GMM on your image to approximate some level of rescaled intensity values for your images, then reversing them such that when the image intensity values are input the probability of membership is output. Use the membership probability as input to your model.
4. Perform a Gaussian smooth on numeric derivative over the image for "approximate edge" location (or use Sobel or whatever) and then perform the above transformations and use as inputs to your network.

I think that idea 1 is your best bet for good classification on very sparse data. You are engaging the "curse of dimensionality" and if you don't have lots of compute resources and lots of training data then you can get results that are pretty meaningless.

Answer 4

How did you choose this particular network architecture? Why don't removing a layer or reducing hidden neurons?

Maybe you can also try to reduce the problem dimensionality using PCA (or something similar) and/or trying with unsupervised methods (clustering methods, SOM). You can find a lot of good material about Computer Vision on Internet.