# Model causality: graphical models and PCA

If we build a graphical model (DAG) we (may) interpret the arrows as causal dependences.

If we build a graphical model based on the variables returned by principal component analysis (PCA) we should obtain a totally disconnected graph (assuming normality). We could interpret this as the variables given by the principal components having no causal dependency and being attributed to different (latent) causes.

How are these interpretations of the data compatible ? Is the concept of causality dependent on the set of variables chosen as a bases?

ps: for easiness of interpetation we may suppose that all variables have the same unit of measure.

1. The principal components you obtain are orthogonal (uncorrelated) among themselves by definition (of PCA), which is already not a great start if you want to assume some kind of dependence among them.
2. Besides that, they do not carry a natural meaning from the perspective of your domain of study (only statistical) except the one you give them.

Maybe you can include principal components in a DAG, but in practice you will have to make several assumptions and in general I really do not think is a good idea (I am very much open to counterexamples). On the other hand, you could infer relationships (correlation, non causation) among the original variables computing the correlation coefficient between them and the principal components. From the interpretation of these correlations potentially you could add new nodes (your principal components) in your original DAG as latent variable with arrows into the variables they mostly correlate with.

To answer this part of the question:

Is the concept of causality dependent on the set of variables chosen as a bases?

The concept of causality depends mainly on you and your situation. Causal inference can be very useful, but I think its reasoning is not unique (except in some straightforward cases). As follows a partial and succint explanation of why (in my opinion):

First: Can I find a DAG for all situations? The concept of causality is always highly dependable on the researcher knowledge and the data that are available. On top of these two things, building a DAG is dependent on your research question. Unless you are looking into a deterministic problem (for example in physics), you cannot build a "complete" DAG the stands in any possible situation (any research question) involving these variables.

The book What If by Hernan and Robins is a simple introduction* to the DAGs world using a perspective from structural causal inference, where you can find some not too complicated mathematical proofs of their reasoning. Second: using my own DAG. The theoretical framework described in the book provides you with some ready-made statistical tools to investigate your data based on the DAG you define, but eventually even in this case I almost always end up using different statistical tools (such as PCA) to back up my reasoning and get new insights.

Just so you know, this relationship between correlations and orthogonality is not widely accepted here

*There are other views and styles you might like better, but in this book they really take it slow and eventually you get quick comprehensive picture on several topics. On the other hand, if you are more machine learning oriented I recommend this.

• Thanks a lot for your answer (+1) and for the references. I will wait a couple of days to see if there are other contributions and if not I will accept your answer. In the meanwhile I will read the first chapter of the book you suggested ;). Ah slow is good :D Feb 15 at 10:16
• Great! I really hope it helps. Consider that other causal inference source use much more the do calculus, but in that book is never used. Not an essential info honestly, but just to inform you. Feb 15 at 12:01
• "so there can be no arrows from one to another" - why would this be true? Variables can be orthogonal / not correlated, yet still be causally related, cf. the discussion linked to in my answer. Missing edges in causal DAGs imply "no direct causal effect", which is distinct from "not correlated". Feb 15 at 17:26
• Yes, I modified the answer so hopefully now is also clearer. However, the point is that we are talking about Principal Components, not any "real" variables, so in this case uncorrelation has a very strong importance per se (in my opinion). Feb 16 at 6:20
• Hmm... but the PCs are just linear combinations of the original ("real") variables, and if those follow a complex causal structure then so will the PCs, no? Sure, you can assume that these particular linear recombinations are not causally related, but that seems to me like the counterpart to assuming that spurious correlations indicate causality. Feb 18 at 10:32

PCA makes no statements about causality; it merely identifies (linearly) uncorrelated factors of variation. That two variables are (linearly) uncorrelated does not imply that they cannot be causally related, so if you want your graphical model to represent causality, then you basically cannot conclude anything about the presence or absence of edges from a PCA.

--

Let us consider a simple example. You have two variables, $$A$$ and $$B$$. $$A$$ follows a standard normal distribution, and $$B$$ is caused by $$A$$ following $$B:=A+\varepsilon$$, where $$\varepsilon$$ follows another standard-normal distribution. Thus, B is also normally distributed with mean 0 and variance 2, and $$A$$ and $$B$$ together are jointly normally distributed, as (I understand) you assumed in your question. If one intervenes on $$A$$, that will change $$B$$, but if one intervenes on $$B$$, that will not change $$A$$.

Firstly, notice that purely from observations of the joint distribution of $$A$$ and $$B$$, it is impossible to conclude anything about their causal relationship! $$A$$ could be causing $$B$$ (as we assume it is), $$B$$ could be causing $$A$$, or both could be caused by a common confounder. (Also see the nice example on Jonas Peters' homepage in this regard.)

Now you perform a PCA, which will give you the principal vectors $$PV_1=(1, 1)/\sqrt{2}$$ and $$PV_2=(1,-1)/\sqrt{2}$$, associated with the principal components $$PC_1=A+B$$ and $$PC_2=A-B$$. The correct causal diagram would now be that both $$PC_1$$ and $$PC_2$$ are caused by both $$A$$ and $$B$$, which follow whatever causal relationship was true for them in the first place. In essence, it seems to me that the principal components will always be descendants of shared confounders, namely all the original variables.

--

Inferring any kind of causal relationships from purely observational data is an extremely hard task and generally impossible without imposing additional assumptions ("I know for a fact that $$B$$ cannot cause $$A$$, and I also know for a fact that there cannot be any third thing influencing both of them."). For a great overview on the assumptions one can make to achieve this, I recommend Jonas Peters' book (PDF available for free under the link). It is, however, very much a topic of current research and essentially unsolved. In this regard, I also found the historical struggle to prove something as "obvious" as that smoking causes cancer quite instructive. ([1], [2], [3], [4])

• Thanks for the contribution. This is an important point I think. Indeed I think the PC algorithm for DAG construction starts from tests of uncorrelation (and partial uncorrelation) to remove edges. This is indeed just a proxy of independence (and partial independence). So here come many questions: Feb 15 at 19:59
• (1) do we need an algorithm testing really independence instead of uncorrelation to to a DAG on PCA variables? (2) if the answer to question (1) is positive, since in the same system one can build a DAG with original variables already at the level of the PC algorithm, this means that a DAG based on PCA variables is probing some different causal relation than the DAG based on the original variables? Feb 15 at 20:00
• (3) In my original question trying to circumvent this issue I wrote the magic word "normality". For normal variables "uncorrelation -> independence", as a well known fact. Does the OP make more sense to you in this setting? Feb 15 at 20:01
• Thanks for the follow up. I will read your answer and see if it addresses the questions. For the moment I send you somr links github.com/huawei-noah/trustworthyAI/tree/master/gcastle Feb 18 at 11:29
• jmlr.org/papers/volume8/kalisch07a/kalisch07a.pdf Feb 18 at 11:30