Let the CDF $F$ equal $1-1/n$ at the integers $n=1,2,\ldots,$ piecewise constant everywhere else, and subject to all criteria to be a CDF. The expectation is
$$\int_{0}^\infty (1-F(x))\mathrm{d}x = 1/2 + 1/3 + 1/4 + \cdots$$
which diverges. In this sense the first moment (and therefore all higher moments) is infinite. (See remarks at the end for further elaboration.)
If you're uncomfortable with this notation, note that for $n=1,2,3,\ldots,$
$${\Pr}_{F}(n) = \frac{1}{n} - \frac{1}{n+1.}$$
This defines a probability distribution since each term is positive and $$\sum_{n=1}^\infty {\Pr}_{F}(n) = \sum_{n=1}^\infty \left(\frac{1}{n} - \frac{1}{n+1}\right) = \lim_{n\to \infty} 1 - \frac{1}{n+1} = 1.$$
The expectation is
$$\sum_{n=1}^\infty n\,{\Pr}_{F}(n) = \sum_{n=1}^\infty n\left(\frac{1}{n} - \frac{1}{n+1}\right) =\sum_{n=1}^\infty \frac{1}{n+1} = 1/2 + 1/3 + 1/4 + \cdots$$
which diverges.
This way of expressing the answer it makes it clear that all solutions are obtained by such divergent series. Indeed, if you would like the distribution to be supported on some subset of the positive values $x_1, x_2, \ldots, x_n, \ldots,$ with probabilities $p_1, p_2, \ldots$ summing to unity, then for the expectation to diverge the series which expresses it, namely
$$(a_n) = (x_n p_n),$$
must have divergent partial sums.
Conversely, every divergent series $(a_n)$ of non-negative numbers is associated with many discrete positive distributions having divergent expectation. For instance, given $(a_n)$ you could apply the following algorithm to determine sequences $(x_n)$ and $(p_n)$. Begin by setting $q_n = 2^{-n}$ and $y_n = 2^n a_n$ for $n=1, 2, \ldots.$ Define $\Omega$ to be the set of all $y_n$ that arise in this way, index its elements as $\Omega=\{\omega_1, \omega_2, \ldots, \omega_i, \ldots\},$ and define a probability distribution on $\Omega$ by
$$\Pr(\omega_i) = \sum_{n \mid y_n = \omega_i}q_n.$$
This works because the sum of the $p_n$ equals the sum of the $q_n,$ which is $1,$ and $\Omega$ has at most a countable number of positive elements.
As an example, the series $(a_n) = (1, 1/2, 1, 1/2, \ldots)$ obviously diverges. The algorithm gives
$$y_1 = 2a_1 = 2;\ y_2 = 2^2 a_2 = 2;\ y_3 = 2^3 a_3 = 8; \ldots$$
Thus $$\Omega = \{2, 8, 32, 128, \ldots, 2^{2n+1},\ldots\}$$
is the set of odd positive powers of $2$ and $$p_1 = q_1 + q_2 = 3/4;\ p_2 = q_3 + q_4 = 3/16;\ p_3 = q_5 + q_6 = 3/64; \ldots$$
About infinite and non-existent moments
When all the values are positive, there is no such thing as an "undefined" moment: moments all exist, but they can be infinite in the sense of a divergent sum (or integral), as shown at the outset of this answer.
Generally, all moments are defined for positive random variables, because the sum or integral that expresses them either converges absolutely or it diverges (is "infinite.") In contrast to that, moments can become undefined for variables that take on positive and negative values, because--by definition of the Lebesgue integral--the moment is the difference between a moment of the positive part and a moment of the absolute value of the negative part. If both those are infinite, convergence is not absolute and you face the problem of subtracting an infinity from an infinity: that does not exist.