I'm training a fully-connected layer with a custom loss $L_1$ to perform dimensionality reduction. This loss is in function of the weights, which pushes the network to a solution which has some properties. In the current setting, I know how to train it through SGD as a stand-alone layer, because given this loss, I know how to update the weights such that it minimizes objective $L_1$. However, in the end I want to use this layer to make classifications, thus I would like to combine $L_1$ with a cross-entropy loss, in a end-to-end fashion. Is it possible to do that?

While a simple fully connected layer will reduce dimensionality, it won't necessarily find a solution whose weights have the same properties driven by this loss. Then, I'd like to combine both: a fully-connected layer, but that also finds a solution with said properties obtained by minimizing this loss.

An alternative interpretation, as loss $L_1$ wants to learn a set of weights for a feed-forward layer and cross-entropy $L_2$ measures the classification performance, maybe $L_1$ should be a regularization term then (as it is only in function of the weights), applied only to the weights in the layers of interest. But, if that's the case and I include it as a regularization term, how do I handle the specific update rule for the weights in this layer (obtained by deriving the loss with respect to the weights)?


  1. Is it possible to combine these two losses in a end-to-end fashion such that the gradients affect each other?
  2. Ideally, this dimensionality reduction layer would precede a prediction layer, thus making some sort of skip-connection. How would that affect my update rule when using such "regularization"? Should I backprop the gradients regarding $L_1$ through the prediction layer as well?


Consider the PCA loss, which may be defined as:


This objective yields the following update rule for the weights $W$:

$$W^{(t+1)} = W^{(t)} + \eta \mathbf{x}\mathbf{x}^T W^{(t)}.$$

As a stand-alone layer, update rule based on gradients $\frac{\partial L_1}{\partial W}$ are used immediately in the fully-connected layer, as shown in this horrible diagram I tried to make below:

Stand-alone linear layer with the PCA objective

Ok, but now I'd like to combine to insert it before the prediction layer, for instance:

Stacked linear layer and the PCA objective. Is it possible to fit them together?

How do I update weights $W$ now? And should I backpropagate the gradients $\frac{\partial L_1}{\partial W}$ to the regular fully-connected layer that precedes the layer I inserted (depicted as the first rectangle in the diagram)? Errata: Now it occurred to me that the second diagram is wrong; the gradients w.r.t. loss $L_1$ should be used to update the second layer itself (which already receives the gradient from the last fully-connected layer). Thus, I guess it should backpropagate to the first layer the regular gradient, isn't it?

Maybe I'm looking at it at the wrong angle, and this layer should only be the last layer. But, then, how would I combine it with cross-entropy?

  • $\begingroup$ I've posted this question to r/machinelearning, but I think I'm looking for a more theoretical answer, thus I cross-posted it here (please, let me know if I did wrong here). However, I'm unsure to which SE site I should post. I hope I picked the right one. $\endgroup$
    – KnownAd
    Nov 17, 2018 at 13:48
  • 1
    $\begingroup$ Loss function is not a layer of neural network, so your question is unclear. You can use the same network, but to estimate parameters use different loss function, in many packages this is just changing single parametr in optimizer. Could you edit your question to give us an example of what you want to achieve? $\endgroup$
    – Tim
    Nov 17, 2018 at 14:01
  • $\begingroup$ @Tim, thanks for pointing that out. I tried to make it clearer. Is it any better? $\endgroup$
    – KnownAd
    Nov 18, 2018 at 0:28
  • $\begingroup$ I can't see what's the problem in here. Loss is not a layer. If you want to reduce dimensionality of your data, then simply insert a fully connected later of expected dimensionality somewhere. There are multi-task networks, but this does not seem to be the case. Cloud you describe in more detail what you are trying to achieve rather then the proposed solution. $\endgroup$
    – Tim
    Nov 18, 2018 at 6:34
  • $\begingroup$ I'm not saying loss is a layer. Yes, a fully connected layer will reduce dimensionality, but not necessarily find a solution whose weights have the same properties a given loss would drive my weights to achieve. I know how I can train it as a stand-alone layer, because given this loss, I know how to update the weights such that it minimizes my objective. I'd like to combine both: a fully-connected layer, but that also finds a solution with said properties obtained by minimizing this loss. Is there a way that I can integrate them and benefit from end-to-end learning? $\endgroup$
    – KnownAd
    Nov 18, 2018 at 15:06

1 Answer 1


I think your network architecture is better understood as:

enter image description here

Gradients should flow back to the dimension reduction layer from both the PCA loss and the cross-entropy loss on the predictions. If you're using a graph-based system like Tensorflow, Theano, Caffe, etc., this should happen automatically. Note that your update for the PCA loss should actually be $$W_{t+1}=W_t + \eta \tilde{\Sigma}W_{t}$$ where $$\tilde{\Sigma}=E[{xx}^T]\approx\frac{1}{b}\sum^{b-1}_{i=0}{x_ix_i}^T$$which is just gradient ascent with the expectation taken over a mini-batch. Alternatively, you could precompute a covariance matrix using all (or a large portion of the very redundant) available data and use $\tilde{\Sigma}$ as a constant. So there is no need for a custom update.

However, let's assume you do want a custom update as represented by a custom gradient (for example, a reduced variance estimate of the gradient). Then you would only need to overwrite the gradient flowing back from the PCA loss into the reduction layer and most graph based programs make this fairly easy to do. If your custom update cannot be understood as an alternative gradient (although most can), then you could use ADMM where you alternate between optimizing with respect to the PCA loss using the custom update and applying gradient descent with respect to the cross-entropy loss. However, since you PCA loss is unbounded from above (unless there are some other constraints on the magnitude of $W$), I'd recommend against this.

  • $\begingroup$ That helped me clarify a lot of things. And ADMM looks very interesting, I think it might come in hand. Thank you for these pointers! Regarding the loss being unbounded from above, that was on point, as I was actually facing a problem regarding it. I was considering adding a $\mathcal{l}_2$-norm constraint to see if that would help. $\endgroup$
    – KnownAd
    Nov 26, 2018 at 11:19
  • 1
    $\begingroup$ L2 norm would work. So would constraining each vector in $W$ to have unit norm and requiring W to be orthonormal. Normalization constraints can be imposed simply by normalizing W before using the matrix for classification. $\endgroup$ Dec 2, 2018 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.