Split Simulated dataset into train and test dataset?

Question

I am working on my thesis, and I wonder how the paradigm with splitting your dataset into training and test dataset works for simulated data.

1. Does it make sense to split a dataset that is simulated into train and test?

My thoughts: The datasets simulated are very controlled in the sense, that the data that are simulated is within a given range, hence the test set will look like the training set. Then you loose the natural variability in data taken from observations in nature. The simulation approach taken is simulatation with Halton sequence and then apply a transformation to get samples within the ranges.

Hope someone can clarify.

Answer 1

Whether a simulation with cross-validation makes sense, very much depends on what your goal is.

Generally, simulating data is a highly recommended step in order to understand and test the behavior of a model under controlled settings. And if you do so, you should try to exactly replicate the analytic framework which you intend to apply to your real data. If it involves cross-validation (e.g. because you care about the generalizability of your model), then you should apply cross-validation to your simulated data as well.

In most cases, simulated data will be more well-behaved than real-world data, e.g. with respect to extreme outliers. So yes, you might overestimate the performance of your model and you should keep that in mind.

For the simulation, try to investigate the properties of your real data as good as possible. That is: which distributions can describe your features? What are the characteristics of the parameters that describe these distributions? (e.g., the range of means and SDs in case you assume normal distributions). Is there a correlation structure between features?

Try to reproduce these characteristics as closely as possible. But again, how much work you may put into this depends on the goal of your simulation. Your goal may simply be a sanity check - show that your model predicts well if you include features that are correlated to your target variable; or show that the model predicts at chance level if you only feed it with noise - in which case a simple procedure for simulated data generation will suffice most of the time. By contrast, if the goal of your simulation is e.g. to compare different models given the specific properties of your data, then you will have to try to reproduce these properties as precisely as possible to draw meaningful conclusions.