This note depends on results contained in Moser's Linear Models: A Mean Model Approach. I'll cite some results from this book in what follows. When I saw your question, I started looking through the book: this note is just the way my thoughts were organized after.
Let $y \sim \mathcal{N}_n(\mu, \Sigma)$ be the response, with $\mu$ containing the fixed effects and $\Sigma$ containing the random effects.
Take $y^T A_i y$ to be the sums of squares corresponding to each term (covariates and interactions) in the model. Note that these sums of squares are invariant to whether terms are fixed or random. Assume that each $A_i$ is symmetric and idempotent, which will be true in most models of interest.
When it holds that $$I = \sum_i A_i,$$ which amounts to the sums of squares corresponding to a decomposition into orthogonal subspaces since we've assumed the $A_i$ are projectors, and $$\Sigma = \sum_i c_i A_i,$$ by Cochran's theorem (lemma 3.4.1),
\begin{equation}
y^T A_i y \sim c_i \chi^2_{d_i} (\mu^T A_i \mu/c_i),
\end{equation}
for $d_i = tr(A_i)$, and $y^T A_j y$ is independent of $y^T A_k y$ for $j \ne k$.
The term $$\tilde{F} = \frac{y^T A_j y / d_j}{y^T A_k y / d_k} \sim \frac{c_j \chi_{d_j}^2(\mu^T A_j \mu / c_j)/d_j}{c_k \chi_{d_k}^2(\mu^T A_k \mu / c_k)/d_k}$$ is indeed a (central) $F$ statistic if and only if
\begin{align*}
\frac{c_j}{c_k} & = 1 , \tag{1} \\
\mu^T A_j \mu & = 0 , \tag{2} \\
\mu^T A_k \mu & = 0 , \textrm{ and }\tag{3}
\end{align*}
When these three conditions are satisfied, we can compute $p$-values corresponding to the statistic $\tilde{F}$. These terms basically just aid in computability since the $c_i$'s depend on variance components and the the noncentrality parameters depend on the mean $\mu$. The second condition ensures that $\tilde{F}$ will have (at least) a noncentral $F$ distribution. Under the second condition, the third condition gives that $\tilde{F}$ has a central $F$ distribution.
The expected mean squares ($\mathrm{EMS}$) corresponding to the $i^\mathrm{th}$ sum of squares $y^T A_i y$ is $$\mathrm{EMS}_i := \frac{1}{tr(A_i)} \mathbb{E} [y^T A_i y] = \frac{tr(A_i \Sigma) + \mu^T A_i \mu}{tr(A_i)} = c_i + \frac{\mu^T A_i \mu}{tr(A_i)},$$ where $tr(A_i\Sigma) = c_i \, tr(A_i)$ due to cor 3.1.2. The ratio $$\frac{\mathrm{EMS}_j}{\mathrm{EMS}_k} = \frac{c_j + \frac{\mu^T A_j \mu}{tr(A_j)}}{c_k + \frac{\mu^T A_k \mu}{tr(A_k)}} = 1$$ if conditions $(1)$, $(2)$, and $(3)$ hold. This is why people inspect the ratio of $\mathrm{EMS}$ when determining which sums of squares to divide to form a $F$ statistic to test a particular null hypothesis.
We use conditions $(1), (2)$, and $(3)$ to specify the null hypothesis. In my experience, when the term (corresponding to $j$) that we're interested in testing is random, we make the null hypothesis be $c_j/c_k = 1$, and, when it's fixed, we make the null hypothesis be $y^T A_j y = 0$. In particular, these amount to us being able to choose $k$ so that the rest of conditions $(1), (2)$ and $(3)$ are satisfied. Such a choice of $k$ isn't always possible, which leads to Behrens-Fisher-like difficulties.
This doesn't explain anything particularly related to the problem at hand, but that just amounts to computing $\mu$ and $\Sigma$. I hope this is seen to be a useful way of thinking about the problem. Note that example 4.4.1 works out what all of the quantities above are in the two-way ANOVA example.
The difference is due to the problem structure and not due to convention. These different approaches (two-way vs repeated measure) change $\mu$ and $\Sigma$, which changes the EMS, which changes which $k$ we choose to construct the test.
Let's consider the model
\begin{equation}
y_{ijk} = \mu_0 + \mathrm{id}_i + \mathrm{Xw1}_{j} + \mathrm{id * Xw1}_{ij} + \mathrm{R(id * Xw1)}_{k(ij)},
\end{equation}
where $i$ denotes the level of $\mathrm{id}$, etc. Here $k$ denotes which of the 3 replicates are being considered.
We now introduce some helpful vector notation: write $y = (y_{111}, y_{112}, y_{113}, y_{121}, \dots y_{20,3,3})$. Since this data is balanced, we can make us of kronecker product notation. (As an aside, I was told that Charlie Van Loan once called the kronecker product "the operation of the 2000s!") Define $\bar{J} \in \mathbb{R}^{m \times m}$ to be the matrix with all entries equal to $\frac{1}{m}$ and $C=I-\bar{J}$ to be the centering matrix. (The centering matrix is so named since, for instance, $\|C x \|_2^2 = \sum_i (x_i - \bar{x})^2$ for a vector $x$.)
With this kronecker product notation under out belt, we can find the matrices $A_i$ mentioned above. The sum of squares correspoding to $\mu_0$ is
\begin{equation}
SS(\mu_0)
= n (\bar{y}_{\cdot\cdot\cdot})^2
= \|(\bar{J} \otimes \bar{J} \otimes \bar{J}) y\|_2^2
= y^T (\bar{J} \otimes \bar{J} \otimes \bar{J}) y,
\end{equation}
where the first component $\bar{J} \in \mathbb{R}^{20 \times 20}$, the second is in $\mathbb{R}^{3 \times 3}$, and the third is in $\mathbb{R}^{3 \times 3}$. Generally speaking, the matrices in those components will always be of that size. Also, the sum of squares due to $\mathrm{id}$ is
\begin{equation}
SS(\mathrm{id})
= \sum_{ijk} (\bar{y}_{i\cdot\cdot} - \bar{y}_{\cdot \cdot \cdot})^2
= \|(C \otimes \bar{J} \otimes \bar{J}) y\|_2^2
= y^T (C \otimes \bar{J} \otimes \bar{J}) y.
\end{equation}
Notice that $SS(\mathrm{id})$ does indeed measure the variation among levels of $\mathrm{id}$. Similarly, the other matrices are $A_{Xw1} = \bar{J} \otimes C \otimes \bar{J}$, $A_{id * Xw1} = C \otimes C \otimes \bar{J}$, and $A_{R()} = I \otimes I \otimes C$.
This is shown to be consistent with aov by running code to give, for instance, the residual sum of squares $SS(\mathrm{R(id * Xw1)}) = y^T A_{R()} y$:
mY <- c()
for(j in 1:(nrow(d)/3)) {
mY <- c(mY, rep(mean(d$Y[3*(j-1)+(1:3)]), 3))
}
sum((d$Y - mY)^2) #this is the residual sum of squares
At this point, we have to make some modeling choices. In particular, we have to decide whether $\mathrm{id}$ is a random effect. Let's first suppose that it isn't a random effect, so that all effects besides the replication are fixed. Then
\begin{equation}
\mathbb{E} [y_{ijk}] = \mu_{ij} = \mu_0 + \mathrm{id}_i + \mathrm{Xw1}_{jk} + \mathrm{id * Xw1}_{ij}
\end{equation}
and $R(\mathrm{id * Xw1})_{k(ij)} \sim_{iid} \mathcal{N}(0, \sigma^2)$. Notice that there's no dependence between distinct observations. In vector notation, we can write $$y \sim \mathcal{N} (\mu, \Sigma)$$ for $\mu = \mathbb{E} [y] = (\mu_{11}, \mu_{12}, \dots, \mu_{20,3}) \otimes \mathbf{1}_{3}$ and $\Sigma = \sigma^2 (I \otimes I \otimes I)$.
Noticing that the sum of all $5$ of the $A$'s defined above is the identity, we know by cochran's theorem that, among other things, $$SS(\mathrm{Xw1}) = y^T A_{Xw1} y \sim \sigma^2 \chi^2_{(19)(1)(1)} (\mu^T A_{Xw1} \mu / \sigma^2)$$ and $$SS(\mathrm{R(id * Xw1)}) = y^T A_{R()} y \sim \sigma^2 \chi^2_{(20)(3)(2)} (\mu^T A_{R()} \mu / \sigma^2)$$ and these sums of squares are independent.
Now, in line with what we discussed above, we want conditions $(1), (2),$ and $(3)$ to hold. Notice that condition $(1)$ holds (because there's no other variance components to complicate things.) What's really cool to notice now is that $\mu^T A_{R()} \mu = 0$, since $\mu$ is constant along this third "component" that is being centered by $A_{R()}$. This means that $(3)$ is behind us. Therefore we only have to fret about condition $(2)$: if we assume it (as a null hypothesis) then we're assuming that $0 = \mu^T A_{Xw1} \mu = \sum_{ijk} (\mu_{ij} - \bar{\mu}_{i \cdot})^2$, which is the same as $\mu_{ij} = \bar{\mu}_{i \cdot}$ for all $i,j$, which is the same as $\mathrm{Xw1}_j = 0$ and $\mathrm{id * Xw1}_{ij} = 0$ for all $i,j$ (since the mean level is in the other terms.)
In summary, the null hypothesis can be seen to just be testing whether a noncentrality parameter is zero, which is equivalent to effects concerning the covariate being zero. The repeated measures case follows a similar line of reasoning, where we instead make the modeling choice that the $\mathrm{id}$ effect is random. There, condition $(1)$ will become the null hypothesis.
Related to the R command, like you mention in the comments to the original post, this error term just specifies which terms are to be considered as random effects. (Note that all terms that are to be included in the model should be plainly input or input inside the Error() term. This is why there's a difference between id/Xw1 = id + id:Xw1 and id being in the Error term. Non-included terms are lumped in with the error in the sense that $A_{R()} + A_{id * Xw1}$ is relabeled as $A_{R()}$.)
Here's the explicit details related to the repeated measures case where the terms related to $\mathrm{id}$ (which are $\mathrm{id}$ and $\mathrm{id * Xw1}$) are random. We'll see that this is the more interesting case.
There we have the same sum of squares matrices (since they don't depend on whether a factor is fixed or random.) The covariance matrix there is
\begin{align*}
\Sigma
& \stackrel{(a)}{=} \sigma^2_{id} (I \otimes J \otimes J) + \sigma^2_{id * Xw1} (I \otimes C \otimes J) + \sigma^2_{R()} (I \otimes I \otimes I) \\
& = \sigma^2_{id} (3)(3) (A_{\mu_0} + A_{id}) + \sigma^2_{id * Xw1} (3) (A_{Xw1} + A_{id * Xw1}) + \sigma^2_{R()} (A_{\mu_0} + A_{id} + A_{Xw1} + A_{id * Xw1} + A_{R()}) \\
& = ((3)(3)\sigma^2_{id} + \sigma^2_{R()})A_{\mu_0} + ((3)(3)\sigma^2_{id} + \sigma^2_{R()}) A_{id} + ((3)\sigma^2_{id * Xw1} + \sigma^2_{R()}) A_{Xw1} + ((3)\sigma^2_{id * Xw1} + \sigma^2_{R()}) A_{id * Xw1} + \sigma^2_{R()} A_{R()},
\end{align*}
where $J$ is the matrix of all ones. The first and last summand on the right hand side of equality (a) offer intuitive explanations: the first summand shows that there's an additional source of correlation among observations with the same $\mathrm{id}$, and the third summand shows, as in the two-way example, the base source of variation. This second summand is less intuitive, but among observations with the same \mathrm{id}, it can be seen as increasing variation between observations with same $\mathrm{Xw1}$ while decreasing variation between observations with different $\mathrm{Xw1}$, due to the shape of $I \otimes C \otimes J$.
Also, since all of the terms related to $\mathrm{id}$ are random, the mean
is just due to $\mathrm{Xw1}$, so that $\mathbb{E} [y_{ijk}] = \mu_{j} = \mu_0 + \mathrm{Xw1}_j$, or $\mu = \mathbf{1} \otimes (\mu_1, \mu_2, \mu_3) \otimes \mathbf{1}$.
Notice that, related to condition $(1)$: we have $$\frac{c_{Xw1} }{c_{id * Xw1}} = \frac{(3)\sigma^2_{id*Xw1} + \sigma^2_{R()}}{(3)\sigma^2_{id*Xw1} + \sigma^2_{R()}} = 1,$$ while $$\frac{c_{Xw1}}{c_{R()}} = \frac{(3)\sigma^2_{id*Xw1} + \sigma^2_{R()}}{\sigma^2_{R()}} \neq 1.$$ Further, related to condition $(3)$ both $\mu^T A_{Xw1*id} \mu = 0$ and $\mu^T A_{R()} \mu = 0$. Also, related to condition $(2)$: we see that
\begin{align*}
\mu^T A_{Xw1} \mu
& = \|A_{Xw1} \mu\|_2^2 \\
& = \|(\bar{J} \otimes C \otimes \bar{J}) (\mathbf{1} \otimes (\mu_1, \mu_2 \mu_3)' \otimes \mathbf{1}) \|_2^2 \\
& = (20)(3) \|C (\mu_1, \mu_2 \mu_3)'\|_2^2 \\
& = (20)(3) \sum_j (Xw1_j)^2.
\end{align*}
Therefore, if the denominator sum of squares was the residual $\mathrm{R(id * Xw1)}$ like before, there would be both conditions $(1)$ and $(2)$ in the null hypothesis---since those are the two conditions that aren't satisfied without assumptions. However, if we were to use denominator sum of squares as the interaction, since condition $(1)$ is already satisfied, the null hypothesis would just be condition $(2)$. So, as you mention in your question, these different denominators just amount to different null hypotheses.
This analysis technique we use allows the choice of which null hypothesis is being tested to be transparent. Indeed, we can see this by writing out the conditions mentioned in the previous paragraph more explicitly. Using the denominator as the residual sum of squares forces us to test $Xw1_j = 0$ for all $j$ and $\sigma^2_{id * Xw1} = 0$, while using the denominator as the interaction sum of squares allows us to simply test $Xw1_j = 0$ for all $j$.
subjecteffect (and all its interactions!) as random, whereas 2-way ANOVA treats it as fixed. I have to think more about it to figure out all the details. $\endgroup$