(Since it is not specified, I am going to assume that the $Z_t$ series is IID with zero mean and fixed variance, as is normally the case in these models.) This time series will be non-stationary if $\mathbb{E}(X_t) \neq 0$ for any $t$. If that is the case then you are correct that the mean will approach zero asymptotically, and therefore the process will be non-stationary. However, if $\mathbb{E}(X_t) = 0$ then obviously the asymptotic convergence to zero would not entail a loss of stationarity.
It is a common misconception in time-series analysis to think that a root inside the unit circle is a sufficient condition for stationarity (see here for further discussion). The recursive equation is insufficient to determine stationarity without setting a "starting distribution" for the process. For stationary AR(1) processes, we have $|\phi| < 1$ but we also usually stipulate that $Z_t \sim \text{IID N}(0, \sigma^2)$ and we stipulate the starting distribution:
$$X_0 \sim \text{N} \Bigg( 0, \frac{\sigma^2}{1-\phi^2} \Bigg).$$
In this case the marginal distribution of the process is equal to the starting distribution at every time index, and so the process is stationary. (In fact, it can be shown that it is strongly stationary.) It is common for texts on time series analysis to specify only a recursive equation for the series, and fail to specify the starting distribution that "anchors" the distribution. In these cases, absent some specification to the contrary, it is usual to take the stationary starting distribution as an implicit aspect of the model.