5
$\begingroup$

I need for verifying that the training set I'm using to build a model is equally distributed to the test set.

The model is for prediction purposes and I think it is necessary the 2 samples are equally distributed; in such sense, it is needed to test that the sample on which you build your model (training set) is distributed as the sample on which you backtest the model (test set).

Usually, in credit scoring/rating model the population stability index is used for this purpose, but it does not seem to be a robust statistical measure.

In your opinion, could I use the K-S two-sample test?

Alternatively, can you suggest some other statistical measure or test better than that one to check for the distribution stability?

$\endgroup$
  • 1
    $\begingroup$ If the training and test data are randomly partitioned from a single data source, then you're absolutely certain they have the same distribution without doing a test. $\endgroup$ – dsaxton Jan 8 '16 at 15:03
  • $\begingroup$ Thanks for the comment @dsaxton, but I did not select the training and the test set randomly. It is a time series dataset, so I divided the sample up 2 sub-samples on timely basis; the training set consists of the ~70% oldest data and the test set of the ~30% recentest data. This is the reason why I need a test/statistics. $\endgroup$ – Quantopik Jan 12 '16 at 15:50
  • $\begingroup$ So do you assume that your time-series is non-stationary? $\endgroup$ – usεr11852 Feb 1 '16 at 7:21
3
+50
$\begingroup$

First, about measuring the fit. The Kolmogorov–Smirnov test is for a one dimensional distribution. Though it was extended to multivariate data, it wasn't designed for time series. I'm not sure how do you use your time series data. If you are interested in just the probability of an event to happen, you can use the test. However, note that you will lose all information like "Event B always comes after event A" and that might be where the gold is.

Going a step backwards, you said that you used your old data as the train and the new data as your test. You can split you data in different ways and avoid the problem in the first place.

In case that you are interested in predicting future behavior, you can split each time series into past (train) and test (future). Note that you can choose a different point for any series. This way you can train on that that now is considered to be your future and still get a valid estimation.

Some time it is required that all the series will be split in the same point in time. In this case you might consider creating few data set (e.g., one ending in January, one ending in February, etc).

The advantage here is that you will be able to estimate how good your model is as time changes. Note that while it is likely that the underling distribution will change, your model is probably looking for a narrower aspect and might be more robust.

You might be coping with a problem of concept drift (or time related domain adaptation). Reading some survey on these topics might give you some useful idea.

$\endgroup$
  • $\begingroup$ +1 @DanLevin! Thanks you too for the answer. Yes, I'm interested only on the event probability and I will do other tests/analysis to check for the causality. At this step, I'm interested in the data distribution only. $\endgroup$ – Quantopik Feb 1 '16 at 10:11
5
+50
$\begingroup$

In the limit as the quality of your random sampling method approaches perfect randomness, and in the limit as number of samples approach infinity, then the distributions of the training and testing samples will become identical.

But after considering your comment to dsaxton, it appears that you are in a special case where you are dealing with time series problems.

Usually with common learning tasks when the time of the sample arrival is ignored, it is implied that all samples occur at the same time. Therefore, the PDFs of the training samples are assumed to be the closest estimation to the PDF of the testing set (since they all occur at the same time). In this case, random sampling is your friend.

But, since you are not assuming the simplistic assumption above, instead you are acknowledging the fact that testing instances are necessarily those that appear after the training samples (which is a more realistic assumption), then it is part of your time series problem that your model must deal with the fact that the PDF of the arriving samples changes as a function of the time.

Therefore, when dealing with time series problems, you must not eliminate the shift/change in PDF as time passes across the training and testing sample sets. Instead, take it as a challenge to identify how well your model is adopting to the fact that the PDF is shifting/changing over time.

If you eliminate such challenge from the evaluation by constructing training and testing set that maintain the same PDF (despite the time shift), then you are essentially performing an evaluation that does not show how useful your prediction model is in time series problems.

Alternatively, you can consider that time series problems are a special case of domain adaptation problems, where the domain variation is caused by variations in the time.

So in summary the answer is: you must not ensure identical/similar PDFs between training and testing samples, because it is a primary objective of your model to adapt to the fact that the PDF is shifting by time.

Q: In your opinion, could I use the K-S two-sample test?

A: No.

Q: Alternatively, can you suggest some other statistical measure or test better than that one to check for the distribution stability?

A: Yes. Do nothing about it. If you wish more samples to better identify the PDF shift as a function time, then this is another problem (where you can find out amount of training samples that you model needs less/more samples compared to other models).

$\endgroup$
  • $\begingroup$ +1 Thanks for the answer @caveman! Can you explain more in detail the reason why I cannot use the KS test, please? $\endgroup$ – Quantopik Jan 31 '16 at 23:54
  • $\begingroup$ Well KS 2-sample test essentially tells whether two samples have identical/similar distributions, right? If you use this to ensure that your training and testing samples of identical/similar distributions, then you are implying that there is no change in the distribution of old and new samples, which means that change in time is irrelevant! Then why are you using time series to begin with? Time series are used when you think that the distributions are shifting as a function of time. You MAY use this test to do the opposite: show that distributions are changing! (maybe better tests exist). $\endgroup$ – caveman Feb 1 '16 at 0:44
0
$\begingroup$

If you selected training and test sets randomly from your data, you shouldn't have to worry about equal distributions.

More importantly, you test your model, which you generated from the training set, on the test set and by doing this you verify that you can use your model to predict other values than the ones from the training set. If this works, you won't have to worry about the sets being equally distributed.

$\endgroup$
  • $\begingroup$ Please, take a look to my comment below the question. Anyway, thanks for the answer 😊 $\endgroup$ – Quantopik Jan 12 '16 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.