I have figures below taken from Diganta Misra's paper "Mish: A Self Regularized Non-Monotonic Activation Function." They visualized both the loss output landscapes in Fig 4 below:
And they also showed different activation functions landscapes:
Questions: What is a landscape please for either a loss function or activation function? Second, how do we get the 2D coordinates for the activation function landscape to draw it with these colors and how do we get the 3D coordinates from the loss output landscape?
For example, if we have a 5-layered architecture network with 3 activation functions and a sigmoid function layer for classification. Sigmoid outputs 2 numbers given that we have 2 classes, then we have a binary loss function to make things simple. How do we take numbers if possible from this toy network to visualize loss output landscape and activation function landscape?
