2
$\begingroup$

A "logarithmic neuron" is defined as follows [1]:

enter image description here

Which for inputs $\left\{ {{x_1},...,{x_n}} \right\}$ yields an output of $z=\prod\limits_{i = 1..n} {x_i^{{w_i}}}$ (in MATLAB, the activation function is considered a separate layer, so I'm going to ignore it from now on).

I'm trying to implement a layer of such neurons as a MATLAB class that inherits from nnet.layer.Layer (which is how custom layers should be defined so that they are compatible with the Deep Learning Toolbox), but my present difficulty is deriving expressions for the derivatives of the loss function through the layer.

The loss function is defined as:

$$ {L_{SPES}} = \frac{1}{2}\left( {\sum\limits_{i = 1..m} {{{\left( {\frac{{t - y}}{t}} \right)}^2}} } \right) \tag{1} $$

so to my understanding, the backward propagated derivative of the loss through the output (regression) layer is therefore:

$$ \frac{{\partial L}}{{\partial y}} = \frac{{y - t}}{{{t^2}}} \tag{2} $$

Proceeding to the logarithmic layer - the forward pass can be defined like this,

$$ z = \exp \left( {\sum\limits_{i = 1..n} {{w_i} \cdot \ln \left( {{x_i}} \right)} } \right) = \prod\limits_{i = 1..n} {x_i^{{w_i}}} \tag{3} $$

and the article mentions that the equation for weight updates in this layer is:

$$ \frac{{\partial L}}{{\partial {w_{ji}}}} = \frac{{\partial L}}{{\partial {y_j}}}\frac{{\partial {y_j}}}{{\partial {v_j}}}\frac{{\partial {v_j}}}{{\partial {w_{ji}}}}\mathop = \limits^? {y_j}\sum\limits_k {{\delta _k}{w_{kj}}{y_i}} \tag{4} $$

My questions are:

  1. How can I express $\frac{{\partial L}}{{\partial {\boldsymbol{W}}}}$ as $f\left( {\frac{{\partial L}}{{\partial Z}},\boldsymbol{X,Z,W}} \right)$?
  2. What should be the expressions for $\frac{{\partial L}}{{\partial \boldsymbol{X}}} = g\left( {\frac{{\partial L}}{{\partial Z}},\boldsymbol{X,Z,W}} \right)$?
$\endgroup$
  • $\begingroup$ Welcome to CV! Since this question is primarily software related, it might be better suited on stackoverflow. $\endgroup$ – Frans Rodenburg Oct 31 '18 at 3:34
  • $\begingroup$ @Frans Thanks for the comment! Actually I have considered this aspect before posting, and have reached the conclusion that the "math" part is the major part of the problem (I should be able to handle the implementation once the math is out of the way). I'll think how to make this aspect clearer in the question. $\endgroup$ – Dev-iL Oct 31 '18 at 7:11
1
$\begingroup$

I'm not entirely sure how to write the solution in "mathematical notation" since it involves permuting the matrices in several dimensions (i.e. "multidimensional transpose"), so I'm posting the code I ended up using, including a brief introduction.


Let the dataset consist of 5 dimensions as follows:

  • Dimensions 1 and 2 are singletons (supposed to represent spatial dimensions, byproduct of using functions meant for images on series).
  • Dimension 3 represents the various inputs; originally "channels" in images).
  • Dimension 4 represents the number of the observation in the mini-batch.
  • Dimension 5 represents the number of the neuron inside the layer.

For the logarithmic layer:

  • Weight initialization:

    layer.Weights = 4*rand(1,1,nInputs,1,nNeurons)-2;
    
  • Forward pass (or Prediction):

    Z = permute(prod(X .^ layer.Weights, 3), [1 2 5 4 3]);
    
  • Backward pass:

    dLdX = sum(permute(dLdZ .* Z, [1 2 5 4 3]) .* layer.Weights ./ X, 5);
    

    dLdW = sum(permute(dLdZ .* Z, [1 2 5 4 3]) .* log( X ), 4);
    

For the SPES regression layer:

  • Forward loss (between the predictions Y and the training targets T):

    loss = 0.5 * sum( ( 1 - Y./T).^2, 3:5 ) / size(Y,4);
    
  • Backward loss:

    dLdY = (Y-T) ./ T.^2;
    
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.