When an ML model (neural network or a regression etc.) is built, is there any way of understanding the logic of the model, i.e., the relationship between the covariates and output in terms of data-generating models, i.e., structural causal models, and therefore understanding the causal relationships rather than the distribution of observed variables, i.e., the weights/beta coefficients of each covariate to the output? Typically in a business setting, root cause analysis is a common topic of discussion and the use of causal discovery/inference would help to answer such questions however neural networks lack this type of interpretation and inference.

Has there been any progress in attempting to use machine learning to create accurate SCMs from output data of neural networks etc?

  • $\begingroup$ For recurrent neural networks you would have to "unpack" the recurrent function compositions so that the result is a directed acyclic graph. A similar comment can be made for 'full' convolutional neural networks which can take in variably-sized inputs. $\endgroup$
    – Galen
    Commented Apr 22, 2022 at 17:57
  • $\begingroup$ @AgnesianOperator. Thank you. Is there any articles/papers on this? $\endgroup$
    – Tom
    Commented Apr 22, 2022 at 18:16

1 Answer 1


Are you familiar with methods of interpreting neural networks which construct "relative variable importance" scores from neural network weights? The basic logic is that NN weights are analogous to the coefficients in a regression model, and can be aggregated to calculate the influence of each individual input variable on the network output. Basically you take the product of the connection weight pairs across each hidden neuron, and add together the products corresponding to a particular input variable.

One approach I like, which uses the connection weight approach, can be found in Olden & Jackson (2002, paper linked below). In addition to calculating the relative variable importance, they create 1000 neural networks from a randomly permuted version of the data, and compare with the "real" model to determine the statistical significance of the variable importance scores and the connection weights. This solves the problem of optimization stability, in which many accurate neural networks, with very different weights and thus variable importance scores, can be created from the same data, and with the same architecture. They reject the idea, commonly held, that neural networks are a black box which can predict with high accuracy but cannot be interpreted.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.