Almost all statistics implicitly condition on N. We treat N as a constant that can come out from the expression $\mathbb{E}\left[\frac{1}{N}\sum_{i=1}^{N}{x_i}\right]$, for example. For that to be appropriate, N has to be a fixed value, which we get by conditioning. Without conditioning on N, as you said, we'd need to know the distribution of N, which is kind of strange to think about ("what are my chances of selecting each value of N for my study?").
It is important to assume that the $x_i$ are independent of N, however. This might be violated if, for example, a study has $100,000 to spend on its subjects and will spend it all no matter how many people are involved. If you are looking at the impact of this spending, then the spending-per-person declines as the number of people increases and N is correlated with your treatment (i.e., spending-per-person).