Language Modelling using Neural Networks

Question

I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus with thousands of sentences, I could make a Bag of Words model out of it. However, I don't know what to do next or how to make the network output probabilities for a given sequences of words.

Answer 1

Neural network language models usually take their input in the form of one-hot encoded vectors. This means that the dimensionality of the vector is equal to the size of the vocabulary and that the vector is all zeros except for a single one at the position that corresponds to the id number for that word. The one-hot input vectors are multiplied by an matrix that converts the inputs to low-dimensional embeddings.

The first generation of neural network language models took as input the previous n-words and used them to predict the next word. The current generation of neural network language models uses something called recurrent neural networks. You can read more about those elsewhere.

For the output you need to produce a vector the size of the vocabulary. Then you use a softmax activation function to turn that vector into a probability distribution over the vocabulary.

Answer 2

Neural language model or another model takes numbers as input and they do not process/work directly over the string (or text ) so you need to convert them into the required format.
One common approach is using bag of words model that you can create by counting the number of times a word is coming into the sentence and making a one hot vector with dimension equals to whole vocabulary. Simply you can use CountVectorizer of sklearn in Python to do this all work for you.
from sklearn.feature_extraction.text import CountVectorizer
Another better approach is making vector of each words using word2vec or Glove. You can use pretrained vectors or you can train your own if your data size is large. You can generate word vectors of each word using spacy library that is beautiful and come with internal feature of generating vectors. To use it you can follow following syntax.

from spacy.en import English
parser = English()
parser.vocab['word'].vector

You can combine different word vectors into a fixed dimension for a sentence and then can get ready to apply the algorithms for getting outputs.

Coming to your main question how to get output probabilities. You can start with basic recurrent neural network language learning model.
You may need a deep learning library like Tensorflow / Keras. In Tensorflow let me tell you code then explain it

    cell = tf.contrib.rnn.GRUCell(state_size) 
    rnn_outputs , final_state = tf.nn.dynamic_rnn(cell,rnn_inputs,seq_len,dtype=tf.float32)
    with tf.variable_scope('softmax'):
        W = tf.get_variable('weight',[state_size,num_classes],initializer=tf.constant_initializer(0.0))
        b = tf.get_variable('bias', [num_classes],initializer = tf.constant_initializer(0.0))


    logits = tf.matmul(final_state, W) + b
    preds = tf.nn.softmax(logits,name='poiu')

First you have to define a cell there are many types like LSTM, GRU and other variants. These are actually used to store information and solve RNN training problems of vanishing and exploding gradients, which in turn makes it possible to learn long-term relationships among sentences.

Next step is to find the output after your data has been passed through a neural network.

After that you define a trainable variable that network uses to save the various learnt parameters that can be used for prediction and all.

Final step is to pass all the learnt parameters through a softamx layer that is used to calculate what is the probability of each sentence belonging to a class.

Note: After all this you have to use/define a loss function that is actually through which networks learn.

loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits,labels="your output"))

You can search over internet for various resources related to this.
https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words http://karpathy.github.io/2015/05/21/rnn-effectiveness/
http://r2rt.com/ # For tensorflow implementation guide