Independence and conditional independence in OLS

In showing that OLS provides a conditionally unbiased estimate of error variance, namely that $\operatorname{E}(\hat{\sigma}|X) = \sigma$, where $$Y = X\beta + \epsilon, \quad \epsilon_i \in N(0, \sigma^2)$$ you get the terms $\operatorname{E}(\epsilon_i \epsilon_j | X)$ in a summation which are suppose to disappear unless $i = j$. By the usual regression model assumptions, $\epsilon \perp\!\!\!\perp X$ and all $\epsilon_i$ are independent. So, $\operatorname{E}(\epsilon_i | X) = \operatorname{E}(\epsilon_i) = 0$ as well as $\operatorname{E}(\epsilon_i \epsilon_j) = \operatorname{E}(\epsilon_i) \operatorname{E}(\epsilon_i) = 0$.

However, how does this imply $\operatorname{E}(\epsilon_i \epsilon_j | X) = \operatorname{E}(\epsilon_i \epsilon_j)$? After all, it's not generally true that if random variables $X$, $Y$ are i.i.d., $X \perp\!\!\!\perp Z$ and $Y \perp\!\!\!\perp Z$ then $XY \perp\!\!\!\perp Z$. (For a counterexample, take $X, Y$ Bernoulli with 0.5 probability of 1, and $Z = X + Y \operatorname{mod} 2$.)

Is a (stronger) conditional independence assumption $\operatorname{E}(\epsilon_i \epsilon_j | X) = \operatorname{E}(\epsilon_i | X) \operatorname{E}(\epsilon_j | X)$ required, or am I missing something?