Some of the confusion regards difference between $e$ and $\epsilon$, and that seems to have been addressed adequately in the comments and other answer(s). But additional confusion expressed by the OP concerns the nature of randomness itself in this context, and in the related issue of the meaning of $E(\epsilon | X)$. Here is an answer that clarifies these issues.
Consider a classic example: $Y$ = son's adult height, $X$ = father's adult height. Suppose $E(Y | X = x) = \beta_0 + \beta_1 x$ is true. Since this is a model for how data might appear, we need some conceptual framework for where/when/how the data are collected. Suppose, for the sake of concreteness, that we are talking about a "typical" sample of people living in the world today, one that is reasonably representative of this human spectrum.
The question of "randomness" can be best understood as something that is unrelated to the actual data; which instead can be understood in terms of "potentially observable data" for the conceptual data collection framework. Given a particular father whose height is 180 cm, but who is otherwise generic within the sampling framework, there is a distribution of potentially observable son's heights. Thus the $Y$ in the expression $Y | X = 180$ can be described as "random" at this stage, having some probability distribution of potentially observable values.
(Note that the "population" of the world is irrelevant in this context - instead, the regression model views the heights of people in the world today as themselves but one of many possible realization of possible heights that could have existed at this particular point in time. One reason the "population" framework makes no sense is that there is no data in population from which to construct the population conditional distributions: How many fathers on the planet have height between 79.9999999...........9 and 80.0000..........1 centimeters? The answer is "none" if you let the "..." run on long enough.)
Now, $\epsilon = Y - (\beta_0 + \beta_1 x)$, which is the difference between the potentially observable (random) $Y$ and the mean of the distribution of such potentially observable $Y$ for the given $x$. The "randomness" in $\epsilon$ is inherited from the "randomness" in $Y$ ( the conditional mean $\beta_0 + \beta_1 x$, while uncertain in the mind, is scientifically fixed in this context).
To understand the condition $E(\epsilon | X=x) = 0$, consider again $X=180$. Here, $\epsilon$ is the deviation of a potentially observable $Y$ for which $X=180$, from the mean of all such potentially observable $Y$. The mean of all such $\epsilon$'s is 0 precisely because the mean of all such $Y$'s is $\beta_0 + \beta_1 (180)$.
By the way, the assumption $E(\epsilon | X=x) = 0 $ is not needed here: it is a mathematical consequence of the more intuitive assumption $E(Y | X = x) = \beta_0 + \beta_1 x$, which simply states that the regression mean function is correctly modeled.