One way of thinking about this statement is to consider that the potential outcomes occur earlier in time than the treatment assignment does. Before treatment is assigned, each individual has two potential outcomes. The assignment to treatment essentially "opens the door" of one of those potential outcomes, and that is the one that then becomes the observed outcome. The only causes of the potential outcomes are covariates $X$. Treatment can't cause potential outcomes because potential outcomes occur prior to treatment; rather, treatment reveals potential outcomes (only one per individual).
If those same covariates $X$ that cause the potential outcomes cause selection into treatment, there is an association, or dependence, between treatment assignment and the potential outcomes. This is what we call confounding. The assumption of unconfoundedness means we have observed all $X$ that are sufficient to eliminate confounding (i.e., all variables that yield an association between the potential outcomes and treatment).
I think it would be far less confusing to write $Z \perp \{Y(0),Y(1)\}|X$. That prioritizes the treatment as being causally separate (i.e., independent) from the potential outcomes. This is not a statement about predicting counterfactuals; it's a statement about the treatment assignment mechanism. In particular, it says the treatment assignment mechanism is independent from (i.e., not associated with) the potential outcomes given $X$. It is this statement about the treatment assignment mechanism that allows us to estimate the treatment effect using only the observed outcomes, the treatment, and the covariates, even though the causal claim we want to make involves only the potential outcomes.