Generally, most deep learning (DL) models are considered opaque black boxes, and post-hoc explanations may not be satisfactory to users, especially for use cases in the legal or clinical world where users may need to then justify their belief in the DL model to other experts. While it's exciting to think about discovering completely unexpectedly important predictors, most likely, explanations would be most trusted if it aligns with the domain knowledge/mental model of users. So I started thinking about how one could incorporate "plausible" explanations derived from domain knowledge into the DL model.

For simplicity's sake, say the primary task is binary classification, as are the set of explanatory tasks. The output of the last layer would be compared to the primary task and the outputs of the second-to-last layer would be compared to the explanatory tasks.

  1. Is it proper to simply add (with some weighting schema) the losses from the primary task (last layer) and the explanatory tasks (intermediate layer) and backprop or do losses based on intermediate layers require special treatment?
  2. Would you "buy" the proposition that for any given sample, the output of the second-to-last layer multiplied by their respective weights in the last layer is a measure of feature importance of the respective "plausible explanations"?
  • $\begingroup$ Can you clarify this statement: " The output of the last layer would be compared to the primary task and the outputs of the second-to-last layer would be compared to the explanatory tasks." -> How are you planning on comparing them? E.g. by adding a fully connected layer on the second-to-last layer for each of the explanatory tasks? $\endgroup$
    – GR4
    Commented Mar 26, 2021 at 8:47
  • $\begingroup$ Ah I should've been more clear, sorry. I was thinking simpler than that. The very last layer would be the typical final single node for binary classification (sigmoid > BCE loss). The outputs of each of the second-to-last layer would be tied to an explanatory task, so the width of the second-to-last layer (which is a fully connected layer) would be the number of explanatory tasks. And in the example where all explanatory tasks are binary classification tasks, sigmoid would be applied the second-to-last layer, and the values would be compared to the true explanatory labels. $\endgroup$
    – Abraxas Yu
    Commented Mar 26, 2021 at 15:23

1 Answer 1

  1. Yes, you can simply add the losses. That being said, since you only want to have the final layer depend on the explanatory tasks, you might as well first fully train your network on the explanatory tasks. Then you could just run a simple logistic regression for your final tasks.
  2. If you think about it, your last fully-connected layer is equivalent to a logistic regression on a set of variables (your explanatory task variables). Logistic regressions are indeed considered a good resource in case you want to have explainable outcomes, because the weights give indeed a rudimentary explanation. That being said, the whole reason for using neural networks is to avoid the need of hand-crafting explanatory tasks to start with. You should ask yourself: can you define all then necessary explanatory tasks that will allow you to train the network optimally on your downstream task in the final layer? (This would depend on your particular tasks, but often it is not the case.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.