How are activation functions calculated in quantized Neural Networks I seem to be missing how activation functions are calculated in a fully integer quantized Neural network.
I understand that when performing inference, the input tensor is scaled to the closest calculated uint8 as shown here.
What I can't follow is how a relu or a sigmoid follow this scaling or how they get modified for this input.

Here for example, a sigmoid roughly makes sense to get an input of between (-5,5), and the relu input for some similar range.
I do not see anywhere how/if this gets scaled to match the input transformations.
When I print the details of a quantized tensor I get this sort of output
'name': 'sequential/conv2d/Relu;sequential/conv2d/BiasAdd;sequential/conv2d/Conv2D;sequential/conv2d/BiasAdd/ReadVariableOp/resource', 'index': 14, 'shape': array([ 1, 26, 26, 12], dtype=int32), 'shape_signature': array([-1, 26, 26, 12], dtype=int32), 'dtype': <class 'numpy.int8'>, 'quantization': (0.007534660864621401, -128), 'quantization_parameters': {'scales': array([0.00753466], dtype=float32), 'zero_points': array([-128], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}} 

Which indicates that the layer that undergoes a Relu operation has a scaling factor, but I don't see how this is applied.
Any thoughts?
Thanks. P.
 A: I know this question is old, but I'm also working on this and will try to give an answer anyway.
Looking at tensorflow lite source code, in particular at the reference implementation of the sigmoid and the tanh activations internal/reference/integer_ops/tanh.h lines 62 and 94 we can see that the quantized activation functions are calculated by lookups to a table which can be found in kernels/internal/common.h line 409.
Also these functions are symmetric so the lookup table only contains values for the x>0 side from which the rest can be derived.
Both tanh and sigmoid use the same LUT because tanh(x) = 2 * sigmoid(2*x) - 1.
The values of the table themselves I imagine are calculated by scaling the function to the range of the data type (in this particular case uint16 (0-65535) and sampling values at regular intervals. The numbers of samples determines the precision, in the case of the table linked above it's 256 samples.
For what regards ReLU i don't think there's any problem with using quantized values, we can still use the same function: ReLU(x) = x < 0 ? 0 : x
Hope this is useful,
Good luck
