Comparison of time series models I'm trying to create a model for a series $X = \{X_1, X_2, ...\}$. I don't assume that the $X_i$ are identical distributed nor that they are independent but at least that they have something in common - so what I mean with that is that for example $X_i$ are all normal distributed but with different means.
I want to predict the distribution of the next value $X_i$ given that I have the older values  $x_{i-1},..., x_{i-k}$ i.e. I want to get  $F(x; x_{i-1},..., x_{i-k}) = P(X_i \le x \mid x_{i-1},..., x_{i-k})$. 
For example one model could be (if I assume that the $X_i$ are normal distributed with some unknown parameters): $F = \Phi_{mean(x_{i-1},..., x_{i-k}), sd(x_{i-1},..., x_{i-k})}(x)$ where $\Phi_{\mu,\sigma}(x)$ is the CDF of the gaussian distribution with mean $\mu$ and standard deviation $\sigma$, $mean$ the sample mean and $sd$ the sample standard deviation.
Now what I'm trying to do is to compare this models but the problem is that for every $X_i$ I just have one sample $x_i$ and the predicted distribution is changing with every $X_i$ (of course).
So what I did is I took $y_i = F(x_i)$ for every $x_i$ and check whether these $y_i$ are uniform distributed, so I basically did a Kolmogorov-Smirnov test to check whether these samples were drawn from an uniform distribution (because the CDF should be uniform distributed) and comparing the p-values of the tests for the different models.
I'm really not sure if this is the correct way to do this but this was my first idea and my intuition says if the model doesn't work well (i.e. doesn't predict the real distribution) the the $y_i$'s shouldn't be uniform distributed but for example there should be more values $\le 0.5$.
So my questions are:


*

*Is this a good way to do this - and when it is, could you explain why in particular it is good (or bad). Has this method what I described even a name?

*How would you compare the models?

*Is there a way to get an absolute measure about how good the model is (for example you can be sure with x% probability that this is the correct model) - not just compared to the other models.


P.s.: Please excuse if I made some mistakes in the notation as you may have notice I'm not an expert in this topic but I think you will get the idea what I want to do ;-)
 A: "Comparing models" needn't involve (and in many situations, probably shouldn't involve) hypothesis testing.
Note that trying to use the CDF as a transform will only work if you know the parameters ... but you don't. So you simply wouldn't get a uniform when you transform.
In general the only level of certainty you can get that you have the correct model is 0% certainty (that is, you can nearly always be completely certain you have not identified the correct model). The point of a model is not to reproduce a phenomenon in its entirety -- we don't have access to enough information to reliably identify all the effects and their form (and usually, can never hope to) -- but instead to describe the most salient features while abstracting out/approximating the remainder. As it is put in Box and Draper:

Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful
  --   George Box & Norman R. Draper, Empirical Model-Building and Response Surfaces

Before you get to the point of comparing models, however, you need to be able to clearly formulate them.
There are numerous time series models that you might find make sense for your situation, but at the least you should be aware of how the common ones are written and understood, and what the "tools of trade" are for identifying, estimating, diagnosing and predicting time series.
To that end you should review some basic work on time series models and forecasting.
There's a good, free (to read online) book - Hyndman and Athansopoulos, Forecasting Principles and Practice - that doesn't go heavily into theory here (scroll down for the links into the book chapters).
You may also find the free to download*  of Shumway & Stoffer's Time Series Analysis, 3E  here of some use.
* (this is from Stoffer's own staff web pages; there's the ordinary and "EZ" versions) 
Both of these books use the free statistical software R.
