You are indeed right that the Nadaraya–Watson estimator would tend to the average since
$$
{\widehat {m}}_{h}(x)={\frac {\sum _{i=1}^{n}K_{h}(x-X_{i})Y_{i}}{\sum _{j=1}^{n}K_{h}(x-X_{j})}}
$$
also can be written as
$$
{\widehat {m}}_{h}(x) = \arg\min_{\beta_0(x)\in\mathbb{R}} \sum_{i=1}^nK_h(x-X_i)(\beta_0(x)-Y_i)^2.
$$
That is, we are locally fitting a constant term at each $x$. It is the bandwidth $h$ what shows how local we are: the larger $h$, the more points are taken into account. Hence, as you said, as $h\to\infty$, ${\widehat {m}}_{h}(x)$ tends to the average of $Y$ for each $x$.
A natural extension of this is to consider fitting something more flexible than a constant. In particular, a locally linear case would be
$$
{\widehat {m}}_{1,h}(x) = \arg\min_{\beta_0(x)\in\mathbb{R}} \sum_{i=1}^nK_h(x-X_i)(\beta_0(x)+\beta_1(x-X_i)-Y_i)^2.
$$
(See equation (5.4) in your referred book for a closed-form solution.) Now it at each $x$ fits a model with a constant and a linear term, where again it is $h$ what shows how "locally" we look at $X_i$: the larger $h$, the less smooth the fitted line will be. In particular, as $h\to\infty$, we take all the points with equal weights and indeed get the OLS line. That is what is discussed at your referred book, page 117.
Here's an illustration of my answer: