Partial correlation between $X$ and $Y$ is obtained by regressing $X$ on $Z$ to obtain $\hat{X}$, regressing $Y$ on $Z$ to obtain $\hat{Y}$, and calculating
\begin{equation}Corr(X-\hat{X}, Y-\hat{Y}).\end{equation}
Why this definition coincides with $Corr(X,Y|Z)$ for multivariate normal distribution. What is the logic behind this statement.
If the set of random variables follows the multivariate normal distribution, the conditional correlation coefficient of a pair of component variables, given all the other variables, is equal to the partial correlation coefficient of the pair.
In partial correlation, you get $\hat{X}$ and $\hat{Y}$ estimated by each $Z$, and center X and Y using different $\hat{X}$ and $\hat{Y}$ for each $Z$. Then you calculated correlation on these centered $X$ and $Y$ (residue) pooled across different $Z$.
In conditional correlation, more precisely you should write this way $E_Z[Corr(X,Y|Z)]$. So for each value Z, you take a slice and calculate $Corr(X,Y|Z)$, calculating correlation incorporates centering by mean, which is essentially the estimate $\hat{X}$ and $\hat{Y}$ for a given value of $Z$ (which means your predictor is a constant). The final step of calculating expectation pool the correlations from all the slices together.
The only difference is that, in partial correlation, your estimation is linear, while in conditional correlation it can be anything.
For multivariate normal distribution, they are proved to be the same. But if the variables are not normally distributed, this coincidence fails.
https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-842X.2004.00360.x
It is a special case for multivariate Normal distribution. Say, if $(y, X) \sim N(\mu, \Sigma)$, then $y|X \sim N(aX+b, \Sigma_{y|X})$.
on the other hand, if you run a regression of $y$ on $X$, you will get the linear relationship as $\hat{y} = \hat{a}X + \hat{b}$. The coefficients that can minimize the expectation of L2 distance between $y$ and $\hat{y}$ is $a$ and $b$, as you already know the underlying distribution.
