You are correct that with a greedy target policy $\pi$, $\pi(s_t, a_t)$ always equals either $1$ or $0$. This does not mean the algorithm cannot learn though. It only means that the algorithm can only learn from the sequence of steps up until an action was taken that the target policy (which can be greedy) would never take, because only from that point on you start multiplying by $0$.
In Section 4, it is described that the sum of all updates that occur during an episode can be written as:
\begin{equation}
\sum_{t=0}^{T-1} \alpha (\bar{R}_t^\lambda - \theta^T \phi_t)\phi_t c_t,
\end{equation}
where:
\begin{equation}
c_t = \sum_{k=0}^{t} g_k \prod_{j=k+1}^{t} \rho_j
\end{equation}
Suppose you want the target policy $\pi$ to be the greedy policy. Suppose, for example, that we took actions that match the greedy action at $t=0$ and $t=1$. This means that $c_0 > 0$ and $c_1 > 0$. Suppose that we took a non-greedy action at $t=2$. Then, from $t=2$ onwards, every $c_t$ will involve a multiplication by $0$, and therefore equal $0$. However, we still have non-zero $c_0$ and $c_1$, so the sum of all updates during the entire episode still consists of non-zero terms for $t=0$ and $t=1$, and we can still learn something from those steps.
This obviously does still mean that learning with a greedy target policy is often quite slow though, because sooner or later you'll have a multiplication by $0$ and you'll be unable to continue learning from that sequence of actions (that is: you'll be unable to continue learning more about the value of the first state-action pair in that sequence. You can still start learning again about another state-action pair, treat it as the beginning of a new sequence).
This is not a problem just with a greedy target policy though. The method has high variance whenever the target policy and the policy used to select actions are very different from each other. Whenever the policies are very different from each other, the multiplications can rapidly get very close to $0$ if you keep playing actions that are unlikely according to the target policy, or rapidly become way too large if, by chance, you happen to keep playing many actions that were unlikely according to the policy $b$, but coincidentally are all really likely according to the target policy.
The assumption that Sean was talking about holds for the policy used to select actions (the policy $b$), this assumption is not required for the target policy $\pi$. $b(s_t, a_t)$ needs to be nonzero for any $s_t$, $a_t$ pair, because otherwise you get a division by zero.