I have figures below taken from Diganta Misra's paper "Mish: A Self Regularized Non-Monotonic Activation Function." They visualized both the loss output landscapes in Fig 4 below:

enter image description here

And they also showed different activation functions landscapes:

enter image description here

Questions: What is a landscape please for either a loss function or activation function? Second, how do we get the 2D coordinates for the activation function landscape to draw it with these colors and how do we get the 3D coordinates from the loss output landscape?

For example, if we have a 5-layered architecture network with 3 activation functions and a sigmoid function layer for classification. Sigmoid outputs 2 numbers given that we have 2 classes, then we have a binary loss function to make things simple. How do we take numbers if possible from this toy network to visualize loss output landscape and activation function landscape?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.