1
$\begingroup$

in skip-gram model we have for arbitrary value of k (window size) and every word occurence in corpus k-1 training examples, what if we could take each word, count output words occurences, transform it with softmax and then amount of training examples would be equal to amount of distinct words in corpus... what am I missing here? or maybe it's correct optimization

example: "She ruminates on finding morals in everything around her. The Queen of Hearts dismisses her with the threat of execution and she introduces Alice to the Gryphon, who takes her to the Mock Turtle." (from https://en.wikipedia.org/wiki/Alice%27s_Adventures_in_Wonderland)

for word "her" and k = 5 we would normaly have 12 training examples, and with idea described above we would have one

import string
import numpy as np
from io import StringIO
from collections import Counter
q = '''She ruminates on finding morals in everything around her. The Queen of Hearts dismisses her with the threat of execution and she introduces Alice to the Gryphon, who takes her to the Mock Turtle.'''
qq = q.translate({ord(c):'' for c in string.punctuation})

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(tokenizer=lambda x: x.split(), ngram_range=(3, 3), token_pattern='word')
cv1 = CountVectorizer(tokenizer=lambda x: x.split(), ngram_range=(1, 1), token_pattern='word')


cv.fit(StringIO(qq))
cv1.fit(StringIO(qq))
vocab = list(cv1.vocabulary_.keys()) # here we obtained vocabulary of corpus
del cv1

grams_with_her = filter(lambda x: x.startswith('her') or x.endswith('her'), cv.vocabulary_.keys())

words = sum(map(lambda x: x.split(), grams_with_her), [])
ngram_training_examples_x = np.zeros((12, len(vocab)))
ngram_training_examples_y = np.zeros((12, len(vocab)))
ngram_training_examples_x[:, vocab.index('her')] = 1
for i, word in enumerate((word for word in words if word != 'her')):
    ngram_training_examples_y[i][vocab.index(word)] = 1
# now we have 12 examples 2x[vocab.len]
ngram = (ngram_training_examples_x, ngram_training_examples_y)
# optimized
cc = Counter(words)
cc.pop('her')

optimized_training_examples_x = np.zeros((1, len(vocab)))
optimized_training_examples_x[:, vocab.index('her')] = 1

optimized_training_examples_y = np.zeros((1, len(vocab)))

values = np.array(list(cc.values()))
softmax_sum = np.exp(values).sum()
for k, v in cc.items():
    optimized_training_examples_y[0, vocab.index(k)] = np.exp(v)/softmax_sum
# and now we have one output example instead of 12
optimized = (optimized_training_examples_x, optimized_training_examples_y)
print(optimized)
print(ngram)

Maybe I just missed something but this would be a great optimization

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.