Why do we typically characterize a stochastic process by its mean and covariance? How else could we? I have three questions below. Please also feel free to point out flaws in my understanding!
A time series of $N$ samples $(x_1, x_2, \cdots, x_N)$ is a realization of some discrete-time stochastic process $\{X_t\}$ indexed by $t(=1, 2, \cdots, N)$. In this case, $(X_1, X_2, \cdots, X_N)$ is a random vector that is fully described by the $N$-dimensional joint distribution function $F(x_1, x_2, \cdots, x_N)$.
Question 1: Why can't we just talk about the random vector $(X_1, X_2, \cdots, X_N)$ and its joint distribution to characterize the sample that we observed? When and why do we need to resort to the concept of 'stochastic process'?
To fully characterize the stochastic process one would need to specify $F$. But in reality this is too difficult.
Question 2: Is it too difficult to specify $F$? Why exactly?
So we typically resort to 'second-order characterizations'. That is, we specify all (possibly) joint moments of $F$ of order up to $2$. This turns out to be:
(order 1) means: $\mu(t) = E[X_t]$ for all $t$
(order 2) covariance: $\gamma(r, s) = E[(X_r - E[X_r])(X_s - E[X_s])]$ for all $r, s$
If $\{X_t\}$ is a Gaussian process, then a second-order characterization also fully specifies $F$. So then we are done.
Question 3: But, what if $\{X_t\}$ is not Gaussian? What information is lost by a second-order characterization? What are some techniques to retain that information? I have briefly read about copulas and suspect they are useful in this case, since they allow to model the marginal distribution (thus non-Gaussian features like skewness, heavy-tailedness) and dependence seperately. Also heard somewhere that higher-order moments are useful. But I have nothing tangible to follow-up on!
 A: Question 1
Given the finite sample, there is no need to mention stochastic processes at all. Indeed, there is no need to use $X_1, X_2, \ldots, X_N$ to denote the $N$ random variables: we could just as well use $A,B, C, \ldots$ as the names of the random variables. But, you bring up time series which suggests that there is some meaning associated with the subscripts such as $X_i$ is the value at time $i$ and that random variables $X_i$ and $X_j$ are separated in time by $i-j$. In this sense, having the notion of stochastic process is helpful in that a lot of the  notations, machinery, and procedures are already set up for your convenience
Question 2
Joint distributions are messier to specify in general. In many cases, the "nicest" results are those that hold when the random variables are independent (or even independent and identically distributed) which special cases also make the joint distribution very easy to specify. Not so in general.
Question 3
Your "second-order characterization" tells you very little about the distribution(s) except that all the random variables have finite means and finite variances (and covariances) as specified in your "order 1" and "order 2". You have no additional information about the process, just what you call the second-order characterization.
For the case of Gaussian processes, the meaning of Gaussian is not just that the random variables comprising the process (or your set of $N$ random variables) are Gaussian random variables but, more importantly, they are jointly Gaussian random variables. Joint Gaussianity implies marginal Gaussianity but the converse is not true. Thus, knowing the mean function and the covariance function allows you to write down the joint pdf of the $N$ (jointly Gaussian) random variables. But, for non-Gaussian processes, you get very little information about the distribution from the "second-order" characterization.
For more on these notions, read my answer to "Does the autocorrelation function completely describe a stochastic process?" over on dsp.SE
