# Language Modelling using Neural Networks

I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus with thousands of sentences, I could make a Bag of Words model out of it. However, I don't know what to do next or how to make the network output probabilities for a given sequences of words.

Neural network language models usually take their input in the form of one-hot encoded vectors. This means that the dimensionality of the vector is equal to the size of the vocabulary and that the vector is all zeros except for a single one at the position that corresponds to the id number for that word. The one-hot input vectors are multiplied by an matrix that converts the inputs to low-dimensional embeddings.

The first generation of neural network language models took as input the previous n-words and used them to predict the next word. The current generation of neural network language models uses something called recurrent neural networks. You can read more about those elsewhere.

For the output you need to produce a vector the size of the vocabulary. Then you use a softmax activation function to turn that vector into a probability distribution over the vocabulary.

• Can you expound on the previous n-words? If I use the one-hot encoded vectors as inputs, let's say for example [0 1 0 0 0 0] represents 'cat' and then I need to get the probability that the next word is 'and' which is encoded as [0 0 0 0 0 1]. How will I use the sentences in the corpus? Will I have to encode them into a one-shot matrix. Example the sentence is 'cat and dog'. I can product the vectors [0 1 0 0 0 0] for cat; [0 0 0 0 0 1] for and; [1 0 0 0 0 0] for dog. These will be the inputs for my neural network? Right? And then the output vector will be the vector for 'and'? Apr 14, 2017 at 4:20
• In the standard language model you predict the probability of the next word given the previous words. So the vector for 'cat' and 'and' would be the input and the vector for 'dog' would be the output. Apr 14, 2017 at 15:23

Neural language model or another model takes numbers as input and they do not process/work directly over the string (or text ) so you need to convert them into the required format.
One common approach is using bag of words model that you can create by counting the number of times a word is coming into the sentence and making a one hot vector with dimension equals to whole vocabulary. Simply you can use CountVectorizer of sklearn in Python to do this all work for you.
from sklearn.feature_extraction.text import CountVectorizer
Another better approach is making vector of each words using word2vec or Glove. You can use pretrained vectors or you can train your own if your data size is large. You can generate word vectors of each word using spacy library that is beautiful and come with internal feature of generating vectors. To use it you can follow following syntax.

from spacy.en import English
parser = English()
parser.vocab['word'].vector


You can combine different word vectors into a fixed dimension for a sentence and then can get ready to apply the algorithms for getting outputs.

Coming to your main question how to get output probabilities. You can start with basic recurrent neural network language learning model.
You may need a deep learning library like Tensorflow / Keras. In Tensorflow let me tell you code then explain it

    cell = tf.contrib.rnn.GRUCell(state_size)
rnn_outputs , final_state = tf.nn.dynamic_rnn(cell,rnn_inputs,seq_len,dtype=tf.float32)
with tf.variable_scope('softmax'):
W = tf.get_variable('weight',[state_size,num_classes],initializer=tf.constant_initializer(0.0))
b = tf.get_variable('bias', [num_classes],initializer = tf.constant_initializer(0.0))

logits = tf.matmul(final_state, W) + b
preds = tf.nn.softmax(logits,name='poiu')


First you have to define a cell there are many types like LSTM, GRU and other variants. These are actually used to store information and solve RNN training problems of vanishing and exploding gradients, which in turn makes it possible to learn long-term relationships among sentences.

Next step is to find the output after your data has been passed through a neural network.

After that you define a trainable variable that network uses to save the various learnt parameters that can be used for prediction and all.

Final step is to pass all the learnt parameters through a softamx layer that is used to calculate what is the probability of each sentence belonging to a class.

Note: After all this you have to use/define a loss function that is actually through which networks learn.

loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits,labels="your output"))


You can search over internet for various resources related to this.
https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words http://karpathy.github.io/2015/05/21/rnn-effectiveness/
http://r2rt.com/ # For tensorflow implementation guide

• Thanks for the very detailed answer. But I'd like to clarify, I'd have to input the Bag of Words model into the network or the one-hot vectors? I need a model that gives the probability that y follows x using neural networks. Would this mean that I need to input a Bag of Words of the sentences in the corpus and wait for the outputs? Apr 14, 2017 at 6:58
• Follow this link and replace random forest with neural network kaggle.com/c/word2vec-nlp-tutorial/details/… Apr 14, 2017 at 7:00
• I'd just have to replace the corpus with the corpus that I'm gonna use right? Apr 14, 2017 at 7:13
• ya . you have to replace the corpus. Apr 14, 2017 at 7:14
• Wait, I think we're not on the same page? The network calculates the probability of each sentence belonging to a class right? But what I need is the probability that Y follows the word X. Apr 14, 2017 at 7:27