Weak law of large numbers: Let $\{h_i, i = 1, \dots n\}$ be an $m \times q$ sequence of iid random variables with mean $\mu = E[h_i]$ that exists and is finite. Then $1/n \sum_{i = 1}^n h_i \rightarrow \mu$ in probability.
I don't understand why we have exists and is finite. Could you give me an example when the expectation exists but is infinite?
The expectation of a random variable $X: \{\Omega, \frak{S}, \mathbb{P}\}\to \mathbb{R}$ is the Lebesgue integral
$$\mathbb{E}[X] = \int_\Omega X(\omega)d\mathbb{P}(\omega).$$
The Lebesgue integral is constructed in a sequence of steps whereby its domain of application is broadened to encompass an ever wider variety of random variables. The first steps ultimately define the integral for variables with non-negative values: the complications of integrating functions which might oscillate arbitrarily between negative and positive values are thereby avoided. To extend the integral to variables with negative values, decompose them into their positive and negative parts:
$$X(\omega) = X^{+}(\omega) - X^{-}(\omega)$$
where $X^{+}(\omega) = X(\omega)$ when $X(\omega)\ge 0$ and $X^{+}(\omega) = 0$ otherwise; similarly, $X^{-} = (-X)^{+}$. These are readily seen to be random variables, too (that is, they will be measurable). The integral is defined to be the difference
$$\int_\Omega X(\omega)d\mathbb{P}(\omega) = \int_\Omega X^{+}(\omega)d\mathbb{P}(\omega) - \int_\Omega X^{-}(\omega)d\mathbb{P}(\omega),$$
each of which involves a non-negative random variable and therefore the meaning of its integral has already been defined.
At this point conventions may vary. The Wikipedia articles I have linked to declare that the integral is defined only when both the positive and negative integrals are finite. One could, however, allow that the integral is also defined when at most one of the integrals is finite. We could say that it equals "$+\infty$" when the integral of the positive part diverges and equals "$-\infty$" when the integral of the negative part diverges.
In this extended sense of being defined, consider a random variable $X$ with a half-Cauchy distribution. Its probability density function (PDF) $f$ is defined and equal to $0$ when $X\lt 0$ and otherwise equal to $(2/\pi)/(1+x^2)$. Thus $X^{+}=X,$ $X^{-}=0$, and by definition
$$\mathbb{E}(X) = \int_{-\infty}^{+\infty} f(x) dx = \frac{2}{\pi}\int_0^\infty \frac{x dx}{1+x^2} - \int_\mathbb{R} 0 dx.$$
Although the first integral diverges, the second obviously is finite, so we could consider this expectation to be infinite. This example answers the question, but a full appreciation requires analysis of a distribution that looks infinite but actually cannot be defined at all. The standard example is the Cauchy distribution (also known as the Student t with one degree of freedom).
For a Cauchy-distributed variable the PDF is $(1/\pi)/(1+x^2)$ everywhere. Splitting the expectation into its positive and negative parts yields
$$\mathbb{E}(X) = \frac{1}{\pi}\int_0^\infty \frac{x dx}{1+x^2} - \frac{1}{\pi}\int_{-\infty}^0 \frac{-x dx}{1+x^2}.$$
Now both sides diverge. Since an expression like "$\infty - \infty$" is nonsensical, we have no choice but to declare this expectation undefined. One way to convince yourself of this is to consider the various ways in which the integral might be calculated: they concern how the limits of $\pm \infty$ are approached. Pick any nonnegative real value $\alpha$. As a mechanism to control the relative rates at which those limits increase, define
$$f(n) = \sqrt{(1+n^2)\exp(2\pi\alpha)-1}.$$
As $n$ grows large without bound, so does $f(n)$. Therefore, if this integral indeed had a well-defined value, it would be valid to compute it as
$$\frac{1}{\pi}\int_{-\infty}^{+\infty} \frac{x dx}{1+x^2} =\,(?) \lim_{n\to\infty}\frac{1}{\pi}\int_{-n}^{f(n)} \frac{x dx}{1+x^2}$$
because both the limits, $-n$ and $f(n)$, are expanding to encompass the entire Real line.
This plot of the PDF shows how $f$ is chosen to assure that the upper limit $f(n)$ extends just a little further to the right than the lower limit $-n$ extends to the left. The parts between $-n$ and $n$ balance, contributing $0$ to the expectation. The value of $f$ is chosen so that the contribution from the excess--shown in red--is always equal to $\alpha$, no matter what $n$ may be.
But a straightforward calculation gives
$$\frac{1}{\pi}\int_{-n}^{f(n)} \frac{x dx}{1+x^2} = \frac{1}{2\pi}\log(1+x^2)|_{-n}^{f(n)} = \frac{1}{2\pi}\left(\log(1+f(n)^2) - \log(1+n^2)\right)=\alpha.$$
(Using the integration endpoints $-f(n)$ and $n$ shows that $-\alpha$ is a possible value of this limit, too.) Accordingly, since this integral can be made to equal any Real number merely by varying how the limits are taken, it cannot be considered to have a definite value.
There are some useful and well-known distributions which have undefined mean. I think one of the most used is Cauchy distribution. See Wikipedia article.
For example the standard Cauchy distribution has pdf $$ f(x)=\frac{1}{\pi(1+x^2)}$$ So, the mean is $$ E(x) = \int_{-\infty}^{\infty}x\frac{1}{\pi(1+x^2)}dx = \frac{log(1+x^2)}{2\pi}\bigg|_{-\infty}^{\infty}$$ You can see that both ends evaluates to infinity, so the expected value is undefined.
