In a Mathematical Model we measure the sensitivity of the output with respect to the parameters and it is desirable that a small change in a parameter doesn't lead to wild fluctuations in the output of the model.

When a model's inputs use different dimensions or units we are advised to use the 'relative sensitivity'. So where $ y = f(x) = (x_1, x_2, x_3 ....,x_n) $, the relative sensitivity of $y$ with respect to $x_i$ for infinitesimally small changes in $x_i$, is given by

$ \frac{x_i}{y} \frac{\partial y}{\partial x_i} $

If $y$ is the error function of an ANN, and $x_i$ is one of its parameters such as a weight, then this equation looks similar to backprop (which is just the chain rule) except for the term:

$ \frac{x_i}{y}$

For example, imagine we have some error value $E$ and it changes with respect to some weight $W_i$. If we apply relative sensitivity we seem to get an error function which says:

"Scale my rate of change of error according to the relative 
sizes of the error and the weight in question"

$$ \frac{\frac{\partial E}{E}}{\frac{\partial W_i}{W}} = \frac{W_i}{E}\frac{\partial E}{\partial W_i}$$

In a neural network this is how weight updates are made using a standard regularization term:

\begin{equation} w_i \leftarrow w_i-\eta\frac{\partial E}{\partial w_i}-\eta\lambda w_i. \end{equation}

The proposed approach would look like this:

\begin{equation} w_i \leftarrow w_i- \eta\frac{W_i}{E}\frac{\partial E}{\partial W_i} \end{equation}

This looks like an elegant way of doing regularization? Am i correct?

  • $\begingroup$ Elegant or not--that will be in the eye of the beholder--the most salient aspects or this proposal are that (1) it is univariate, not multivariate (it doesn't account for simultaneous changes in parameters) and (2) it is ad hoc, vague, and non-quantitative insofar as it does not determine how much to change the parameters nor does it provide any principles or criteria for doing so. As such it's difficult to conceive of this as being a form of regularization. Is it possible you have left out some important details in your description? $\endgroup$ – whuber Apr 17 '18 at 17:46
  • $\begingroup$ I didn't think simultaneous changes were a problem since Neural Nets do it this way. I have included a comparison to the standard weight decay equation with a reg-term. It changes the parameter according the scaled derivative of the error with respect to the weights. $\endgroup$ – COOLBEANS Apr 17 '18 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.