Specifying the conditional distribution for the response in practice? [duplicate]

In normal linear regression, defining $$Y_i = X_i\beta+\epsilon_i,$$ with $$\epsilon_i$$ c.i.i.d. $$\mathcal{N}(0,\sigma^2)$$, is equivalent to say $$Y|X \sim \mathcal{N}_n(X\beta,\sigma^2I_n)$$. The second seems a better choice for generalization, for instance in GLM's, where the distribution of $$Y|X$$ usually belongs to the exponential family.

In practice, if there are no "physical constraints" in choosing the statistical model, how does one model the stochastic part of a GLM? I used to think that you could do it by looking at the histogram of $$Y$$, but this question What if residuals are normally distributed, but y is not? made me realize I was completely wrong!