Backpropagation in a logarithmic layer of a regression NN

Question

A "logarithmic neuron" is defined as follows ^[1]:

Which for inputs $\left\{ {{x_1},...,{x_n}} \right\}$ yields an output of $z=\prod\limits_{i = 1..n} {x_i^{{w_i}}}$ (in MATLAB, the activation function is considered a separate layer, so I'm going to ignore it from now on).

I'm trying to implement a layer of such neurons as a MATLAB class that inherits from nnet.layer.Layer (which is how custom layers should be defined so that they are compatible with the Deep Learning Toolbox), but my present difficulty is deriving expressions for the derivatives of the loss function through the layer.

The loss function is defined as:

$$ {L_{SPES}} = \frac{1}{2}\left( {\sum\limits_{i = 1..m} {{{\left( {\frac{{t - y}}{t}} \right)}^2}} } \right) \tag{1} $$

so to my understanding, the backward propagated derivative of the loss through the output (regression) layer is therefore:

$$ \frac{{\partial L}}{{\partial y}} = \frac{{y - t}}{{{t^2}}} \tag{2} $$

Proceeding to the logarithmic layer - the forward pass can be defined like this,

$$ z = \exp \left( {\sum\limits_{i = 1..n} {{w_i} \cdot \ln \left( {{x_i}} \right)} } \right) = \prod\limits_{i = 1..n} {x_i^{{w_i}}} \tag{3} $$

and the article mentions that the equation for weight updates in this layer is:

$$ \frac{{\partial L}}{{\partial {w_{ji}}}} = \frac{{\partial L}}{{\partial {y_j}}}\frac{{\partial {y_j}}}{{\partial {v_j}}}\frac{{\partial {v_j}}}{{\partial {w_{ji}}}}\mathop = \limits^? {y_j}\sum\limits_k {{\delta _k}{w_{kj}}{y_i}} \tag{4} $$

My questions are:

1. How can I express $\frac{{\partial L}}{{\partial {\boldsymbol{W}}}}$ as $f\left( {\frac{{\partial L}}{{\partial Z}},\boldsymbol{X,Z,W}} \right)$?
2. What should be the expressions for $\frac{{\partial L}}{{\partial \boldsymbol{X}}} = g\left( {\frac{{\partial L}}{{\partial Z}},\boldsymbol{X,Z,W}} \right)$?

Answer 1

I'm not entirely sure how to write the solution in "mathematical notation" since it involves permuting the matrices in several dimensions (i.e. "multidimensional transpose"), so I'm posting the code I ended up using, including a brief introduction.

---

Let the dataset consist of 5 dimensions as follows:

• Dimensions 1 and 2 are singletons (supposed to represent spatial dimensions, byproduct of using functions meant for images on series).
• Dimension 3 represents the various inputs; originally "channels" in images).
• Dimension 4 represents the number of the observation in the mini-batch.
• Dimension 5 represents the number of the neuron inside the layer.

For the logarithmic layer:

• Weight initialization:

  layer.Weights = 4*rand(1,1,nInputs,1,nNeurons)-2;
  

• Forward pass (or Prediction):

  Z = permute(prod(X .^ layer.Weights, 3), [1 2 5 4 3]);
  

• Backward pass:

  dLdX = sum(permute(dLdZ .* Z, [1 2 5 4 3]) .* layer.Weights ./ X, 5);
  

  dLdW = sum(permute(dLdZ .* Z, [1 2 5 4 3]) .* log( X ), 4);
  

For the SPES regression layer:

• Forward loss (between the predictions Y and the training targets T):

  loss = 0.5 * sum( ( 1 - Y./T).^2, 3:5 ) / size(Y,4);
  

• Backward loss:

  dLdY = (Y-T) ./ T.^2;