First, let's make clear here that the base learners most often don't know about the loss function: they most often do not try to minimize it. The loss function is from the boosting part, the base learners are agnostic to it.
The only way they can be exposed to the loss function is through the surrogate targets we train them on. The problem with Bernoulli I will expose below is stereotypical, but under any non-Gaussian log likelihood you'll get a divergence. If you simply use the residuals, you are not actually using the loss function, unless it was actually Gaussian.
Consider the Bernoulli loss function:
$$\mathcal L(\hat y_1,y) = -\log(\hat y_1)y-\log(1-\hat y_1)(1-y)$$
Having obtained $\hat y_1$, the residuals are $\epsilon_1=y-\hat y_1$.
The residuals simply cannot be Bernoulli distributed, primarily because they are continuous, as $\hat y_1$ is often a proportion (assuming a decision tree base learner), but also that the sum $\hat y_1+\hat y_2$ can exceed $0$ or $1$, since residuals can be negative.
As weak learners display a lot of bias, they could predict first small $\hat y_1$, but then negative $\hat y_2$, making $\hat y_1+\hat y_2<0$.
The converse is also true, where they could make $\hat y_1 \approx 1$, but then $\hat y_2>0$ and you may reach a point where $\hat y_1+\hat y_2>1$.
If the loss function was Normal, then we could keep on fitting to residuals, because they likely would continue to follow a Normal distribution.
The motivation to use the pseudo-residuals based on the gradient is that, for the Normal negative loglikelihood case (which is simply the mean squared error), the simple residuals are precisely the negative gradient of the loss function.
It can be shown that, when you predict the pseudo-residuals, you "travel" in the direction that minimizes the loss function.
If you use direct residuals, however, since they are the negative gradient of the Gaussian loglikelihood, you will optimize the Gaussian likelihood, not your loss function.