As pointed out by the DDPM paper, we can choose to reparameterize the prediction of the mean to prediction of the total noise "εθ is a function approximator intended to predict ε from x" (equation 11). Then, during sampling, why don't we directly remove the predicted total noise from the last step (pure noise), but instead sample images step-by-step?
In theory you can do this, but your image will not be very good. Removing all the noise in one step would be the equivalent of asking your model to recover an image that has been completely corrupted by noise.
All the model will be able to do during this prediction is guess some vague fuzzy features shared by your whole dataset, as that's as far as it could get by optimizing the objective at $t=T$. You would then re-corrupt your model's prediction up until we are just a bit under the level of pure noise we were at to start with. Your model now is taking something other than pure noise as input, so it can be a bit more confident about what image may have been corrupted, so it guesses a few more (still fairly vague and fuzzy) characteristics of the image. Then again we renoise up to just a little bit less, and repeat.
When we are half way through sampling, we have something resembling, for example, a human face corrupted with noise. Hopefully your model has seen a lot of human faces corrupted with noise, and the corresponding real human face, so the model will be able to more accurately guess the original uncorrupted image. You can imagine as we continue to denoise, the model becomes more and more sure of what the true image is, and is able to add more and more fine grained details that it couldn't when it was just given pure noise.
These slides contain some good visuals of the recovered images you get at different time steps.
Your mentioned DDPM is Ho et al's Denoising Diffusion Probabilistic Models (2020), and as you rightly quoted that the reparameterization of the forward process posterior variational mean $\mathbf{\mu_{\theta}}$ using $\mathbf{\epsilon_{\theta}}$ is crucial here for a lot of simplification in terms of the variational training objective $L_{\text{simple}}(\theta)$ arrived at Equation (14). Note $\mathbf{\epsilon_{\theta}}$ is a neural network function approximator with inputs $\mathbf{x}_t$ and $t$ to predict $\mathbf{\epsilon}$ from $\mathbf{x}_t$ at each forward step $t$ where $\mathbf{x}_t=\sqrt{\bar{\alpha_t}}\mathbf{x}_0+\sqrt{1-\bar{\alpha_t}}\mathbf{\epsilon}$ (see Equation (9)) due to some nice property of the forward diffusion process, and there's no single noise learned during DDPM's ELBO training from the final stage's standard Gaussian distribution to the target data distribution.
Therefore once it's trained successfully as implemented by the paper's Algorithm 1, DDPM captures all the crucial noises at each time step during the reverse denoise diffusion process to reconstruct the target data $\mathbf{x}_0$ via iterative sampling as implemented by the paper's Algorithm 2. Your concerned one-shot noise removal by simply adding all noises across all time steps is incorrect, because the learned noise at each timestep $t$ in a diffusion process is specific to predict the distribution $p(\mathbf{x}_t)$ at that timestep $t$ only.
