Say I have a neural network which has 1,000,000 parameters built using 100 features and I wish to understand the underlying data-generating process for how the model arrived at each prediction. Simply looking at the value of the node will not provide any information. I am looking to implement a solution that will help to explain the model either by means of a structural causal model (which shows me the causal relationship between endogenous variables) or via an explanation that is comparable to human reasoning. I am opposed to using SHAP values or feature importance tools, not least because it doesn't measure causality, but because values of the importance of features can be skewed by the effect of multi-collinearity. Removing collinearity (for example using PCA) to rectify the skewness leads to reduced interpretability of my variables (contrary to my original task).
-
1$\begingroup$ "Data generating process" and explaining the predictions are two different things. $\endgroup$– TimCommented Sep 27, 2022 at 12:57
-
$\begingroup$ dl.acm.org/doi/10.1145/3313831.3376447 $\endgroup$– Eike P.Commented Sep 27, 2022 at 12:59
-
$\begingroup$ @Tim which term is applicable here? Sorry for the confusion $\endgroup$– TimCommented Sep 27, 2022 at 13:32
-
$\begingroup$ Why do you think a NN fit with disregard for causality can somehow post-hoc be made sense of or has somehow capture something about the true data generating process? There's of course NN approaches that try to address causality, but that's then usually already considered during the modeling (see e.g. Dragonnet). There's also lots evidence for how giving "causal information" to the model during training would help models (see e.g. the idea to create extra examples using the least necessary counterfactual modification to change the label) $\endgroup$– BjörnCommented Sep 27, 2022 at 13:45
1 Answer
I am afraid that the question you are asking doesn't have an answer that would be satisfactory for you. Causal inference us used for finding causal relations in noisy data. A neural network is a mathematical function. It is deterministic, so it directly causes the result to be what it is. Everybmuktiplication, by every weight, addition of every bias, every other mathematical transformation has some, deterministic, impact on the result, so it “causes” it. Surely some of the parameters have a greater impact than others, but causality is not about the strength of the relationship.
What you could do is you could build a simpler model that approximates the neural network and using it will allow you to understand better how it works and this is exactly what expandability tools like LIME do.
