Suppose we are performing a least-squares multiple linear regression of the form $$ \mathbf{Y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}\,, $$ where $\mathbf{Y}=(y_1,y_2,...,y_N)$ are the response variables, $\mathbf{X}$ is the $N \times P$ design matrix, $\boldsymbol{\beta}=(\beta_1,\beta_2,...,\beta_P)$ are the parameters and $\boldsymbol{\varepsilon}$ is the error.
The leave-one-out cross-validation statistic $CV$ is defined as $$ CV=\frac{1}{N}\sum_{i=1}^N y_i-\hat{y}_{[i]}\,,\tag{1}\label{1} $$ where $\hat{y}_{[i]}$ is the predicted value of $y_i$ obtained by fitting the model to all observations except the $i$th one.
According to various sources—I'm following Rob Hyndman's notation here, but it's also described in e.g. An Introduction to Statistical Learning—a well-known "shortcut" to calculating $CV$ in the context of ordinary least squares without having to re-fit the model $N$ times is
$$ CV=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{y_i-\hat{y}_i}{1-h_i}\right)^2\,,\tag{2}\label{2} $$
where $\hat{y}_i$ is the predicted value of $y_i$ from the "full" fitted model using all observations. Here, $h_i$ is the leverage statistic for observation $i$, which Hyndman writes is equal to the $i$th entry on the diagonal of the "hat-matrix" $\mathbf{H}=\mathbf{X} (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'$.
In the case where $N=P$ and therefore $\mathbf{X}$ is a square matrix, all $y_i=\hat{y}_i$ and all $h_i=1$, and the "shortcut" equation above is undefined. However, $CV$ can still be obtained using equation \eqref{1} above.
Is there a "shortcut" to obtain $CV$ without refitting the model $N$ times like in equation \eqref{2}, but which is defined when $\mathbf{X}$ is a square matrix?