The title "errors in variables" and the content of the question seems different, as it asks about why we do not take into account the variation in $X$ when modelling the conditional response, that is, in inference for regression parameters. Those two preoccupations seems orthogonal to me, so here I respond to the content.
I have answered to a similar question before, What is the difference between conditioning on regressors vs. treating them as fixed?, so here I will copy part of my answer there:
I will try to flesh out an argument for conditioning on regressors somewhat more formally. Let $(Y,X)$ be a random vector, and interest is in regression $Y$ on $X$, where regression is taken to mean the conditional expectation of $Y$ on $X$. Under multinormal assumptions that will be a linear function, but our arguments do not depend on that. We start with factoring the joint density in the usual way
f(y,x) = f(y\mid x) f(x)
but those functions are not known so we use a parameterized model
f(y,x; \theta, \psi)=f_\theta(y \mid x) f_\psi(x)
where $\theta$ parameterizes the conditional distribution and $\psi$ the marginal distribution of $X$. In the normal linear model we can have $\theta=(\beta, \sigma^2)$ but that is not assumed. The full parameter space of $(\theta,\psi)$ is $\Theta \times \Psi$, a Cartesian product, and the two parameters have no part in common.
This can be interpreted as a factorization of the statistical experiment, (or of the data generation process, DGP), first $X$ is generated according to $f_\psi(x)$, and as a second step, $Y$ is generated according to the conditional density $f_\theta(y \mid X=x)$. Note that the first step does not use any knowledge about $\theta$, that enters only in the second step. The statistic $X$ is ancillary for $\theta$, see https://en.wikipedia.org/wiki/Ancillary_statistic.
But, depending on the results of the first step, the second step could be more or less informative about $\theta$. If the distribution given by $f_\psi(x)$ have very low variance, say, the observed $x$'s will be concentrated in a small region, so it will be more difficult to estimate $\theta$. So, the first part of this two-step experiment determines the precision with which $\theta$ can be estimated. Therefore it is natural to condition on $X=x$ in inference about the regression parameters. That is the conditionality argument, and the outline above makes clear its assumptions.
In designed experiments its assumption will mostly hold, often with observational data not. Some examples of problems will be: regression with lagged responses as predictors. Conditioning on the predictors in this case will also condition on the response! (I will add more examples).
One book which discusses this problems in a lot of detail is Information and exponential families: In statistical theory by O. E Barndorff-Nielsen. See especially chapter 4. The author says the separation logic in this situation is however seldom explicated but gives the following references: R A Fisher (1956) Statistical Methods and Scientific Inference $\S 4.3$ and Sverdrup (1966) The present state of the decision theory and the Neyman-Pearson theory.
The factorization used here is somewhat similar in spirit to the factorization theorem of sufficient statistics. If focus is on the regression parameters $\theta$, and the distribution of $X$ do not depend on $\theta$, then how could the distribution of (or variation in) $X$ contain information about $\theta$?
This separation argument is helpful also because it points to the cases where it cannot be used, for instance regression with lagged responses as predictors.