I understand that logistic regression has no analytical solution, but I was investigating the marginal effect in a simple logistic regression and followed this train of thought to the final line. Here it seems we have an analytical solution for $\beta$, so I must have done something wrong?
what is mathematically wrong with this statement?
There's nothing mathematically wrong with your derivation, but the symbols must be interpreted in a particular way.
The conditional probabilities in your statement must mean the model predictions. Said differently, the first equation you write already assumes you have a fit logistic regression models, as the conditional probabilities in your model are the predictions from the model.
It no longer works if you interpret those probabilities in any other way, in particular, as empirical sample probabilities. For example:
1) $P(Y = 1 \mid x = 0)$ may not even be approximatable from your sample, you may not observe any data points at $x=0$.
2) If your $x$ feature is continuous, you may not observe any more than one data point at any value of the feature.
3) There may be no value of $t$ for which you have observed multiple data at both $x = t$ and $x = t + 1$. Even if you use other values for your separation, the same may be true.
4) Even if all the above are not a problem, you've observed many data points at values of $x$ that are spaced by some separation, using different reference points to compute your sample probabilities may lead to different answers, i.e. different values of $\beta$.
The magic of the regression model is that it gets around all these problems. It uses all your pairs of data, with all their spacings, to estimate one value of $\beta$ that best describes them all at once. There really is no way to do this in a simpler manner.
