1
$\begingroup$

I was trying to use masked image modeling in huggingface and I saw ViTForMaskedImageModeling in their documentation but I did not understand how it reconstructs the original image loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction
also, it doesn't reconstruct the original image correctly. It gives me noise.

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k")

num_patches = (model.config.image_size // model.config.patch_size) ** 2
pixel_values = image_processor(images=image, return_tensors="pt").pixel_values
# create random boolean mask of shape (batch_size, num_patches)
bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool()

outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction

reconstructed_pixel_values = reconstructed_pixel_values.detach().numpy()
reconstructed_pixel_values = np.transpose(reconstructed_pixel_values[0], (1, 2, 0))

plt.imshow(reconstructed_pixel_values)
plt.show()

output

$\endgroup$

1 Answer 1

1
$\begingroup$

It looks like ViTForMaskedImageModeling adds an additional decoder layer that doesn't exist in the pretrained model provided at "google/vit-base-patch16-224-in21k". This would mean the pixel predictions are coming from a randomly initialized layer (thus why the reconstructions are just noise)

This warning also appears in the code you provided that supports this:

Some weights of ViTForMaskedImageModeling were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['embeddings.mask_token', 'decoder.0.weight', 'decoder.0.bias']

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.