Moving from support vector machine to neural network (Back propagation)

Question

I'm working with text recognition and currently I'm using support vector machine method. I would like to try with neural network also. I read a few documents about how neural network works, but the theory is quite heavy and I don't know exactly how will it apply to my case. So it would be good if someone can help me to make it clear, especially with the neural network's architecture.

• Currently, in SVM, I have 200 features (divided into 4 main categories), which is used to recognize the text. If I move to neural network, With 200 features, does it mean that I will have 200 neurons in the input layer?

• With 200 features, how will that result in the architecture of the neural network (in term of number of hidden layers and neurons in them)?

• In SVM, I have two class classification (basically, true and false) and multi-class classification (labels), how this difference will apply to the output layer of the neural networks?

And I also have a few general questions :

• What will help to decide the number of hidden layers and the number of neurons inside each hidden layer?
• Does the number of hidden layers relate to the accuracy ?

I'm new to neural network so it would be great if you can explain to me in an understandable way. :) Thank you very much.

Answer 1

When using of neural networks is normal to follow good practises and rules of thumb, you can find some scientific papers (like LeCun et al. here) and tutorials. The choice of the number of hidden neurons and layers it's not easy and it could be critical (this is one of the main drawbacks on the use of NNs), if you have N 200 features (and then 200 input neurons) you don't need 200 hidden neurons, I suggest you to read carefully a good description on how neural networks work (my best one is Neural Networks: A Comprehensive Foundation by Simon Haykin). You can start with a small number (try with 50 neurons) and then you need to explore how the NN works with smaller and larger numbers.

It's the same about the number of hidden layer, nowadays it's quite common to have only one hidden layer but you can also try with larger numbers (I've always used only one layer)

You can have only one output neuron, considering number larger than zero as "true" and otherwise as "false", also this is an arbitrary choice even if it's quite common.

In my experience, you need to explore the parameter space varying the number of neurons, of course you would also try to change activation functions and number of hidden layers but it's up to you.

Anyway, I suggest you to read carefully some introductory works on neural networks.

Answer 2

One of the greatest advantages or neural networks is their ability to extract features on their own. Thus, nearly all applications feed the networks raw (or nearly raw) data. In your case, that means the best input for a neural network would be the text you mean to classify (your 200 features can be used as well, but that would be a little more advanced). In terms of hyperparameters (like the number of hidden layers, number of neurons per layer, etc), in my experience it's best to start at recommended values from some tutorial, and adjust them based on bias/variance analysis.

Deeplearning.ai's course on Coursera is a great introduction to neural networks. If you just need some quick results, there's a bunch of tutorials online (Medium is a good place to look) on text classification.

Finally I should mention there are many types of neural networks. Text classification is best treated with Recurrent Neural Networks or Transformers (stick to the first until you know what you are doing). Best of luck!