Why there is high variance of gradients estimated in the short directions in regression? I was trying to understand Ridge Regression and came across the following excerpt from Hastie et al. in The Elements of Statistical Learning (section 3.4.1, Page 67):

If we consider fitting a linear surface over this domain
(the Y -axis is sticking out of the page), the configuration of the data allow
us to determine its gradient more accurately in the long direction than
the short. Ridge regression protects against the potentially high variance
of gradients estimated in the short directions. The implicit assumption is
that the response will tend to vary most in the directions of high variance
of the inputs.

Could someone please help me understand why there is high variance of gradients estimated in the short directions?
My understanding is that along the short directions, there is less variance in the data. So shouldn't the variance of the gradients estimated be lower along the short directions?
 A: In ordinary least squares regression, the variance of the data in the regressors will end up in the denominator for the expression of the error of the parameters
$$\text{Cov}
(\hat\beta) = \hat{\sigma}^2 (X^TX)^{-1} $$
If the columns in the regressor matrix $X$ are perpendicular (and they are if you use principle components) then the standard error can be expressed as
$$s.e.(\beta_i) \approx \sqrt{\frac{\sigma^2}
{n \text{Var}(X_i)} }$$
So larger variance in the regressor $X_i$ means smaller variance/error in the estimate for the coefficient/slope/gradient.
See also the following image. It gives the same correlated data but with a different variance for the $x$ variable. This changes the slope. If the scale of the $x$ variable is smaller then the slope becomes larger (and also the error of the slope).

A: There's a lot of information in the quoted paragraph, so in this post we dissect each sentence. We start by introducing some notation and initial development. Then, we directly interpret the sentences.
Setup
Define $X \in \mathbb{R}^{n\times p}$ as the centered design matrix, assumed non-random and full-rank with $n>p$. Write the compact SVD $X=UDV^T$. Define $y = X \beta^* + \epsilon$ as the response, where $\beta^*$ is the unknown coefficient. Assume $\mathrm{cov}(\epsilon) = \sigma^2 I_n$ has a spherical distribution.
We also change coordinates to better match the ESL argument. Define $\alpha^* = V^T \beta^*$ and $Z=XV$ so that the response $y=Z\alpha^* + \epsilon$. The new design matrix $Z$ is expressed in $V$-coordinates and $\alpha^*$ is the coresponding coefficient. Throughout we focus on $\alpha^*$ as the coefficient of interest, noting that it differs from $\beta^*$ by an invertible transformation.
Initial development
In this section, we first write the OLS and Ridge estimators and their corresponding standard errors. The OLS estimator satisfies
$$\hat\alpha := (Z^TZ)^{-1}Z^Ty = D^{-1}U^T y, \text{ and } \mathrm{cov}(\hat\alpha) = \sigma^2 D^{-2}.$$
The ridge estimator satisfies
$$\hat\alpha_\lambda := (Z^TZ + \lambda I)^{-1} Z^Ty = (D^2 + \lambda I)^{-1} D U^T y, \text{ and } \mathrm{cov}(\hat\alpha_\lambda) = \sigma^2  (D^2 + \lambda I)^{-2} D^2.$$
Next, we state a helpful interpretation. The variance of the $j$th column $Z_j$ of the design matrix $Z$ is $\mathrm{var}(Z_j) = d_j^2$, for $d_j$ the $j$th entry of the diagonal matrix $D$. Recall that $Z_j$ is the $j$th covariate.
The first sentence
The first sentence reads:

If we consider fitting a linear surface over this domain
(the Y -axis is sticking out of the page), the configuration of the data allow us to determine its gradient more accurately in the long direction than the short.

The sentence is expressing a relationship between the estimability of the $j$th entry of $\alpha^*$ and the covariate variance $d_j^2$. We may formalize this as follows. The minimax lower bound for estimating $\alpha^*_j$ is $$\frac{\sigma^2}{d_j^2}.$$ Notice that this is also the squared standard error of the OLS estimator. The bound shows that $\alpha_j^*$ may be estimated better when $d_j$ is larger, for general parameter values.
Second sentence

Ridge regression protects against the potentially high variance
of gradients estimated in the short directions.

This sentence is meant to contrast the ridge estimator with the OLS estimator, so let's compare the two estimators.
Consider a coordinate $j$ with $d_j \approx 0$ so that the corresponding covariate $Z_j$ has very low variance. By inspect the standard errors above, we see that the OLS estimator $\hat\alpha_j$ will have a very large standard error due to the $D^{-2}$ term. In the same manner, we see that the ridge estimator $\hat\alpha_{\lambda,j}$ has a negligible standard error due to the $D^2$ term. This is the described protection: the ridge estimator in short directions (i.e. components $j$ for which $d_j$ is small) has low variance, unlike the OLS estimator.
We remark in passing that -- without any prior belief -- an estimator for the gradient $\alpha^*_j$ should have its standard error blow up as the direction becomes shorter (i.e. $d_j \to 0$) since in the limit there's no information about the gradient. The OLS estimator does have this behavior.
Third sentence

In sufficiently short directions, the implicit assumption is that the response will tend to vary most in the directions of higher variance of the inputs.

This sentence was not exactly right as stated, so we amended the sentence to include a qualifying statement. The writer believes this comports with the intention of the ESL writers.
The statement describes an informal principle underlying the ridge estimator. We introduce some additional notation to more compactly express the principle. Both the OLS and ridge estimators may be written as $S U^T y$ for some diagonal matrix $S$, so in this section we consider estimators that can be described thusly. We describe $S$ as the balancing term and write its $j$th diagonal entry as $s_j$.
The (unqualified) principle suggests that $d_j > d_k$ implies that $s_j > s_k$. That is, the principle suggests that directions for which the covariate has higher variance should also have larger gradient estimators.
For more intuition about this principle, we now visualize the terms $s_j$ for both OLS and ridge. 
The figure shows a plot of $s$ against $d$; the red curve is OLS and the blue curve is ridge. For OLS, the red curve is decreasing, confirming that directions with smaller variance has larger balancing terms.
For ridge, the blue curve is initially increasing then later decreasing. The hump in the blue curve represents the divide between where the ridge estimator prioritizes shrinkage or prioritizes respecting the data. The blue curve initially is increasing since the degree of shrinkage in ridge is relaxed as the direction becomes more variable. Then, when the direction becomes sufficiently variable and hence enough information is present in the data, the balancing term behaves like the OLS estimator.
This shows that the principle is obeyed when $d_j$ is sufficiently small. We can calculate that sufficiently small means that $d_j^2 < \lambda$.
