2
$\begingroup$

I want to implement the image captioning example that https://keras.io/getting-started/sequential-model-guide/#examples has , for experimentation.

Instead of using their mentioned convnet, I decided to use resnet50 that i found at https://github.com/fchollet/deep-learning-models .

After some modifications , my code looks like:

vocab_size = 10000
max_caption_len = 16

model = ResNet50(weights='imagenet')
model.layers.pop()
output = model.get_layer('avg_pool').output



output = model.get_layer('avg_pool').output
output = Flatten()(output)
output = RepeatVector(max_caption_len)(output) # your newlayer Dense(...)
image_model = Model(model.input, output)

language_model = Sequential()
language_model.add(Embedding(vocab_size, 256, input_length=max_caption_len))
language_model.add(GRU(output_dim=2048, return_sequences=True))
language_model.add(TimeDistributed(Dense(2048)))



model = Sequential()
model.add(Merge([image_model, language_model], mode='concat', concat_axis=-1))
# let's encode this vector sequence into a single vector
model.add(GRU(256, return_sequences=False))
# which will be used to compute a probability
# distribution over what the next word in the caption should be!
model.add(Dense(vocab_size))
model.add(Activation('softmax'))



model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

the next step is to actually start training. I dont know exactly what kind of data it requires.

for model.fit([images, partial_captions], next_words, batch_size=16, nb_epoch=100)

What exactly is next_words ? Is it just 0 for the words that are absent in my vocabulary (1000 words in my case) , and 1 for those present ?

Also, where could I get a dataset that can be used ?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.