4
$\begingroup$

I am studying the usage of knowledge distilling and all the contents found on youtube and papers suggests the use of softmax layer as the last layer because of the temperature value and the probability distribution. I'd like to know, if it means it works only with a softmax layer or can i use a sigmoid layer for binary classification on the teacher side and a softmax on the student side? I haven't found so far any example of code/description of context using a sigmoid layer on the teacher's side

$\endgroup$
0

2 Answers 2

3
$\begingroup$

If you have two classes, their probabilities sum to one and are $p$ and $1-p$ respectfully. That is binary classification algorithms usually leave only a single column with zeroes and ones as prediction target. Sigmoid is just softmax for two classes. Just remember that distilation uses a special case of softmax that uses temperature $T$ hyperparameter

$$ q_i = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)} $$

so you need to divide the logits $z_i$ by temperature first.

$\endgroup$
1
$\begingroup$

As long as you calculate the terms in the cross entropy, i.e. $y_i$ for class $i$ correctly, you can use sigmoid. In the sigmoid one, you'll have one output instead of two.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.