5
$\begingroup$

I want to train neural network with text data(natural language) as input for classification purpose. One way for standardizing text data for neural network is to use N-GRAM/SKIP-GRAM representation which will be in vector form.

Is there any other way apart from n-gram/skip-gram to represent data that can be used as input for neural networks?

$\endgroup$
3
$\begingroup$

I've been also trying to use Neural Networks for text categorization/classification with limited success. I tried to move away from unigram/bigram features (very sparse, very high-dimensional) to dense and much smaller dimensionality representations. I tried LDA (Latent Dirichlet Allocation) and some other feature selection/extraction methods but only got inferior performance compared to sparse unigram/bigram features used in Logistic Regression.

I am well aware of recent papers using Recurrent Neural Networks and other deep learning techniques but they need lots of data and demand significant computing power. While the latter I have, in my application I don't possess lots of data. So I must stick with shallow machine learning methods.

I am very interested to find out what dense and low-dimensional features give at least comparable performance to unigrams/bigrams in a setting when datasets are not large enough for deep learning. I am particularly interested in methods analyzing/mining short text documents.

$\endgroup$
1
$\begingroup$

Neural networks is not the best way for text classification and for good improve you need to train it for a long time. If you just want use the NN read more about RNN and Word Embedding. RNN showed a good results for text classification tasks, but it hard to train for a complex tasks. Basically word Embedding is some input layers in network which transform your word (letter) in multi-dimensional space. The best thing that after long time of training words which have similar meaning would be together in a vector space. For example the can be words Cat, Dog, Mouse and so on. And in NN classification tasks will track all changes between similar words in sentence and put the in the same class. The best way to start with RNN is the original Elman paper Finding Structure in Time where he present his Elman RNN. There are a lot of simple examples and also you can find very simple word embeding for a small group of words. So this is of course a one of the simplest RNN but it will show you some basic ideas behind RNN.

$\endgroup$
1
$\begingroup$

Nevermind... I found the answer here PDF link. Using word-of-bag or word class is also possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.