validating a gaussian process fitted to data I am relatively new to applying Gaussian processes to data. I come from a math background but the most popular literature on it seems to be from a machine learning perspective and not from a stochastic process/measure theoretic perspective.
Anyway, I am asked to fit a Gaussian process to data, i.e. $$GP(x) \sim N(\mu(x),\sigma^2(x)).$$ The model upon which the data is based assumes that for $x_1 \neq x_2$, $GP(x_1)$ and $GP(x_2)$ are independent. In addition, we specify parametric forms to $\mu(x),\sigma^2(x)$. These parametric forms are provided since we have restrictions on how the mean and variance behave across $x$.
I have training data and validation data -- samples of $(x_i,GP(x_i))$ for $i=1,\dots,N$ for training and samples of $(x_j,GP(x_j))$ for $j=1,\dots,M$ for validation distinct from training.
Using the training data, I formulate the negative log likelihood (just product of pdf's) and use MLE to obtain the parameters needed for the mean and the variance functions above.
Now, I want to assess the validity of my model for $\mu(x)$ and $\sigma^2(x)$ by taking each sample from the validation set, applying the transformation $\frac{GP(x_j) - \mu(x_j)}{\sigma(x_j)}$, and plotting the histogram to see if it's $N(0,1)$.
I am not so successful in achieving a nice looking histogram that matches the N(0,1) pdf so far. My question is: is it realistic to expect that one can actually find parametric forms for $\mu(x),\sigma^2(x)$ in order to "validate" the use of a Gaussian Process to fit the data? How does one even check whether a Gaussian Process of any sort can be applied to the data in the first place? (My data set takes on positive and negative values so that's a first step).
 A: ... is it realistic to expect that one can actually find parametric forms for $\mu(x)$, $\sigma^2 (x)$ in order to "validate" the use of a Gaussian Process to fit the data?
Yes, if I take "validate" to mean create a model that fits the data. Let's say my observations all have different values for the covariates then it's always possible to design a "prior" mean function that is arbitrarily close to all observed response values. 
If you're looking to use this model to perform statistical inference or make predictions then this would be a bad idea for obvious reasons.
How does one even check whether a Gaussian Process of any sort can be applied to the data in the first place?
In the absence of any data you just have to know whether you could transform the covariates to numbers and the response to real numbers--where the covariates have finite dimension and the response is a scalar. However, I'm not sure you're really asking this as your questions have arisen after performing model checking. 
Some domain knowledge helps when performing model checking (e.g. is the fitted model plausible/useful) but, while I may have approached it differently, what you've done using a hold-out sample is reasonable. Generally one might also check whether the hold-out sample (approximately) follows the fitted multivariate normal distribution and, in addition, you might perform more extensive cross-validation. However, your check alone has spotted an issue to investigate--perhaps you provide could more information to help us diagnose the problem.
