I read from textbook that 'Autocovariance can fully charaterise the time series' joint distribution', I do not fully understand the connection between covariance and joint distribution here. Please explain to me, someone?
A stationary Gaussian process is completely characterized by the combination of its mean, variance and autocorrelation function. The statement as you read it is not true. You need the following additional conditions:
Then the entire stochastic process is completely characterized by its autocovariance function (or equivalently its variance $σ^2$ + autocorrelation function).
This simply relies on the fact that any multivariate Gaussian distribution is uniquely determined by its mean vector and its covariance function. So given all the conditions that I stated above the joint distribution of any $k$ observations in the time series has a multivariate normal distribution with mean vector having each component equal to $μ$ (by stationarity) each component has variance $σ^2$ (again by stationarity) and the covariance components are given by the corresponding lagged covariances in the autocovariance function (again stationarity comes in because the autocovariance only depends on the time difference (or lag) between the two observations whose covariance is being taken.