How capable are neural nets? Do I need to do feature selection?

Question

I'm super new to Neural Nets and machine learning in general, so bear with me. I guess my question is "How capable are neural nets?" I've worked with the example of training a net to predict if an 8x8 image is a "4" or a "3". Currently, I am feeding in each value into my neural net:

[
1,0,0,1,0,0,0,0,
1,0,0,1,0,0,0,0,
1,0,0,1,0,0,0,0,
1,1,1,1,1,1,0,0,
0,0,0,1,0,0,0,0,
0,0,0,1,0,0,0,0,
0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0
],

Kind of hard to see but there is a '4' in there. I have successfully trained a my neural net to recognize 4's vs 3's pretty effectively. But now I am working with some acceleration data to predict different motions.

I've read that a multiple hidden layers allow you to emulate non-linear functions. But I don't remember enough about linear algebra (not to mention non-linear algebra) for that to make much sense to me. , but then I started to hear about feature selection, and basically creating summary statistics about my data to use as inputs rather than raw values. I understand the need to do that when you get into larger data sets where you would have an 80x80 matrix instead of an 8x8. But is there a reason to do feature selection on smaller data sets in order to give the net more insights about the data? Or does the net able to take into consideration basically all of the nuances of the data?

Answer 1

The answer to "how capable are neural nets" is, in short, "very". It has been shown that neural networks can, given enough structure, approximate any nonlinear function (universal function approximation).

https://en.wikipedia.org/wiki/Universal_approximation_theorem

This says that even a single layer with enough neurons can represent any function. However, the number of neurons needed in a single layer may be very large, and techniques like convolutional nets (CNNs) favor having multiple (in some cases, dozens or hundreds) of layers instead of a single massive hidden layer.

For your problem in particular, it is small enough that it may be easily solved by a network with a single, fully-connected hidden layer with a few dozen neurons. For the output layer, softmax would be the typical approach with one neuron for each digit you want to recognize. For this approach (or any other classifier, for that matter), you would need to feed in enough training data examples (including translations, scalings, and rotations of the digits you're trying to classify, depending on how robust you want the classifier to be to these sorts of permutations) to allow it to learn a broad array of data conditions.

Feature selection has historically been used to effectively reduce the dimensionality of the representation fed into classifiers. There are many techniques that have been used to find low-dimensional representation of data sets as a precursor to classification tasks. However, in recent years, convolutional neural nets with deep learning architectures without any explicit feature selection have proven to out-perform most other methods for state-of-the-art image recognition.

This is not to say that in specific cases you couldn't find specific feature extraction techniques as a precursor to a smaller neural network classifier that would work quite well for your problem, but this requires hand-tuning and trial and error to extract adequate representations that support downstream classification. In some sense, CNN architectures are doing the same thing automatically - they are learning nonlinear feature extractions that best support classification in a low-dimensional space.