Following Hernán and Robins' Causal Inference, Chapter 16: Instrumental variable estimation, instrumental variables have four assumptions/requirements:
$Z$ must be associated with $X$.
$Z$ must causally affect $Y$ only through $X$
There must not be any prior causes of both $Y$ and $Z$.
The effect of $X$ on $Y$ must be homogeneous. This assumption/requirement has two forms, weak and strong:
- Weak homogeneity of the effect of $X$ on $Y$: The effect of $X$ on $Y$ does not vary by the levels of $Z$ (i.e. $Z$ cannot modify the effect of $X$ on $Y$).
- Strong homogeneity of the effect of $X$ on $Y$: The effect of $X$ on $Y$ is constant across all individuals (or whatever your unit of analysis is).
Instruments that do not meet these assumptions are generally invalid. (2) and (3) are generally difficult to provide strong evidence for (hence assumptions).
The strong version of condition (4) may be a very unreasonable assumption to make depending on the nature of the phenomena being studied (e.g. the effects of drugs on individuals' health generally varies from individual to individual). The weak version of condition (4) may require the use of atypical IV estimators, depending on the circumstance.
The weakness of the effect of $Z$ on $X$ does not really have a formal definition. Certainly IV estimation produces biased results when the effect of $Z$ on $X$ is small relative to the effect of $U$ (unmeasured confounder) on $X$, but there's no hard and fast point, and the bias depends on sample size. Hernán and Robins are (respectfully and constructively) critical of the utility of IV regression relative to estimates based on formal causal reasoning of their approach (that is, the formal causal reasoning approach of the counterfactual causality folks like Pearl, etc.).
Hernán, M. A. and Robins, J. M. (2017). Causal Inference. Chapman & Hall/CRC.