Understanding neural network outputs in terms of causal models?

Question

When an ML model (neural network or a regression etc.) is built, is there any way of understanding the logic of the model, i.e., the relationship between the covariates and output in terms of data-generating models, i.e., structural causal models, and therefore understanding the causal relationships rather than the distribution of observed variables, i.e., the weights/beta coefficients of each covariate to the output? Typically in a business setting, root cause analysis is a common topic of discussion and the use of causal discovery/inference would help to answer such questions however neural networks lack this type of interpretation and inference.

Has there been any progress in attempting to use machine learning to create accurate SCMs from output data of neural networks etc?

Answer 1

Are you familiar with methods of interpreting neural networks which construct "relative variable importance" scores from neural network weights? The basic logic is that NN weights are analogous to the coefficients in a regression model, and can be aggregated to calculate the influence of each individual input variable on the network output. Basically you take the product of the connection weight pairs across each hidden neuron, and add together the products corresponding to a particular input variable.

One approach I like, which uses the connection weight approach, can be found in Olden & Jackson (2002, paper linked below). In addition to calculating the relative variable importance, they create 1000 neural networks from a randomly permuted version of the data, and compare with the "real" model to determine the statistical significance of the variable importance scores and the connection weights. This solves the problem of optimization stability, in which many accurate neural networks, with very different weights and thus variable importance scores, can be created from the same data, and with the same architecture. They reject the idea, commonly held, that neural networks are a black box which can predict with high accuracy but cannot be interpreted.

https://depts.washington.edu/oldenlab/wordpress/wp-content/uploads/2013/03/EcologicalModelling_2002.pdf