I want to implement the image captioning example that https://keras.io/getting-started/sequential-model-guide/#examples has , for experimentation.
Instead of using their mentioned convnet, I decided to use resnet50 that i found at https://github.com/fchollet/deep-learning-models .
After some modifications , my code looks like:
vocab_size = 10000 max_caption_len = 16 model = ResNet50(weights='imagenet') model.layers.pop() output = model.get_layer('avg_pool').output output = model.get_layer('avg_pool').output output = Flatten()(output) output = RepeatVector(max_caption_len)(output) # your newlayer Dense(...) image_model = Model(model.input, output) language_model = Sequential() language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len)) language_model.add(GRU(output_dim=2048, return_sequences=True)) language_model.add(TimeDistributed(Dense(2048))) model = Sequential() model.add(Merge([image_model, language_model], mode='concat', concat_axis=-1)) # let's encode this vector sequence into a single vector model.add(GRU(256, return_sequences=False)) # which will be used to compute a probability # distribution over what the next word in the caption should be! model.add(Dense(vocab_size)) model.add(Activation('softmax')) model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
the next step is to actually start training. I dont know exactly what kind of data it requires.
for model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)
What exactly is next_words ? Is it just 0 for the words that are absent in my vocabulary (1000 words in my case) , and 1 for those present ?
Also, where could I get a dataset that can be used ?