1
$\begingroup$

I already trained student-teacher networks using the main idea of knowledge distillation which has a form of (source of image)

enter image description here

I wondered if there is a way to use a different input size (already used to train our teacher) while training the student network. Let me give an example to be more precise. Suppose that we have an image classification problem. While training my large teacher network for this task, my inputs were images of size 1024x1024 pixels. After having my pretrained teacher network, I have again some images for the same problem but with size 512x512 pixels. Since to train my student, I need prediction results of my pretrained teacher, how can it be possible to use my new input and pass through the teacher network?

Not that I also look at a very recent knowledge distillation survey paper (Knowledge Distillation: A Survey). I know that I can make experiments by increasing 512x512 to 1024x1024 or decreasing 1024x1024, but I want to know if using different input sizes in teacher and student networks is possible.

I did some research, but I could not find an example of it.

Improve this question
$\endgroup$
8
  • $\begingroup$ What would be the (technical) problem with it? There is no (technical) problem with using teacher-predicted class probabilities as the target for student training, you just happen to feed differently sized inputs into each. Of course, that might somehow affect performance, but why not try it? Perhaps it's more tricky to use teacher's final hidden layers weights as a supervision signal, because depending on the architecture (e.g. different pooling layers) dimension might not match, but you could use the whole hidden layer as an extra (high-dimensional) output of the network in that case. $\endgroup$
    – Björn
    Commented Sep 13, 2022 at 8:11
  • $\begingroup$ I know that instead of logits we can use other feature maps (maybe attentions) for that. After training a teacher network designed to take an input of size X, I do not understand how to adapt it to train my student, which requires an input of size Y (with X>>Y). I need to pass my Y to the trained network as input but it does not trained (want size X) for that. At that point, my question is not related to the performance results or to how to use a traditional student-teacher network (both of them require input of the same sizes). $\endgroup$
    – Mas A
    Commented Sep 13, 2022 at 9:39
  • $\begingroup$ You just run images of size X through the teacher and get the outputs (e.g. the logits for each class). Then you take the same inputs (but this time re-sized to size Y) and train with the outputs of the teacher as part of the target. While you get the outputs from the teacher, there's no back-propagation or the like, while when you train the student there is (but the teacher output is taken as given). There's really nothing in the traditional student-teacher set-up that requires same sized images. $\endgroup$
    – Björn
    Commented Sep 13, 2022 at 11:06
  • $\begingroup$ Or, alternatively, consider the image in its original size as the input. The resizing to size X or size Y can also be considered as a non-trained first step in each network, for the teacher it's to size X for the student it's to size Y. $\endgroup$
    – Björn
    Commented Sep 13, 2022 at 11:07
  • 1
    $\begingroup$ Well, if you want to place such extreme restrictions on yourself, you won't be able to do this. The only realistic way this will work is, if you resize images (a dataset being available in different sizes is essentially the same thing, just that someone else has done the resizing for you; random cut-outs of size Y from X-sized images is also conceivable). Ideally, you want to start with images that are at least of size max(X, Y), because down-sizing them (or cutting out) is likely a safer operation in how it affects the models than having to increase size. I don't see how else this would work. $\endgroup$
    – Björn
    Commented Sep 14, 2022 at 17:51

1 Answer 1

Reset to default
2
$\begingroup$

We use what is described as a projector which is a learnable layer that maps between the student and teacher models. For a teacher representation z_t and a student representation z_s, the loss can be constructed as: L = d( projector(z_s), z_t) where d is some distance function (L2, etc). By representation, this is typically the representation immediately before the final layer. This would allow you to use your teacher model (which has been trained with a higher resolution) without any upscaling.

If the spatial dimensions are mismatched, you could use a conv layer.

projector() is typically a linear layer but it can be multiple layers. It is then thrown away after training.

As examples:

https://arxiv.org/pdf/2209.02432.pdf

https://arxiv.org/pdf/2303.11098.pdf

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.