GP takes a function space view, meaning it tries to find the function that maps your features to your target(more approximation rather than mapping). The difference to generic models like linear regression is that instead of imposing structure to the function(in linear regression, the imposed structure is the function being linear on its parameters). GP defines a function space and looks for the best fitting function rather than the best fitting parameters to the problem at hand.
GP also assumes that the best fitting function is a random variable, and outputs follow a joint normal distribution. As a side note, this is not the only difference between the two models, they are entirely different, but the point mentioned above is the relevant difference to question.
Since the normal distribution is only characterized by its covariance matrix and mean vector, since mean vector and covariance matrix estimated using kernel function(covariance function), we have the result "covariance function of a Gaussian process induces properties." Meaning depending on the specific kernel function we choose, the properties of distribution, thus the properties of GP change.
In a nutshell, for stationarity, we can define it as: If the properties of distribution do not change over time, then it is stationary. If it does, then it is not stationary. Note that this is not a rigorous definition.
Connection to stationarity comes from kernel function being stationary or not. Meaning if the problem at hand is not stationary, you cannot use a stationary kernel function. Thus the choice of kernel function depends on the problem.
Edit:
Please refer to Chapter 2.2 of Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. Williams.
Below function is function 2.19 in the book:
$\mathbf{f_*}|X_*,X,\mathbf{f} \sim N(K(X_*, X)K(X,X)^{-1}\mathbf{f}, K(X_*, X_*)-K(X_*, X)K(X, X)^{-1}K(X, X_*))$
Where:
$X$: Training data input (features)
$X_*$: Test data input (features)
$\mathbf{f}$: Training data output (target)
$\mathbf{f_*}$: Test data output (target), the value we are trying to predict.
$K(., .)$: Kernel function
The above distribution's mean vector gives your point predictions, and the covariance matrix gives your uncertainty estimates. As you can see from the above formulation, both the mean vector and covariance matrix are functions of the kernel function. Thus, "mean vector and covariance matrix estimated using kernel function(covariance function)".
I believe if you keep reading the book, everything will become more apparent in each chapter.
The specification of the prior is important, because it fixes the properties of the functions considered for inference. Above we briefly touched on the mean and pointwise variance of the functions. However, other characteristics can also be specified and manipulated. Note that the functions in Figure 1.1(a) are smooth and stationary (informally, stationarity means that the functions look similar at all $x$ locations). These are properties which are induced by the covariance function of the Gaussian process; many other covariance functions are possible.
