I'm studying in R a way to generate synthetic data starting from real world data and, in particular, I'm focusing on the year 2008 of the "Household power consumption" dataset (free to download at UCI Machine Learning Repository).

My problem is that the synthetic data are far from real data and I'm can't understand why.

These are the steps that my process has so far:

  1. I imported the data for the year 2008 and I aggregate them on hourly basis.
  2. I'm started to focus on the less related variable in the dataset (voltage) and I find all the components using the command decompose(). I removed the noise effect and seasonality (I will add them later).
  3. I divided the data in different seasons and I started from the winter, checking the histogram and I found that a normal distribution could be a good fit form my data:

The red line is a <code>dnorm(x, mean = mean(w, na.rm= T), sd = sd(w, na.rm = T))</code>

  1. So I generated my data using fit_w <- rnorm(length(w), mean(w, na.rm = T), sd = sd(w, na.rm = T)), but the final results are not so satisfying: Difference between data

My question is: how can I generate data in a way that they will look similar to the real dataset? Because all my efforts and strategies that I tried until now will lead to similar results.

  • $\begingroup$ There's clear periodic effects/autocorrelation in your data that are not in your model. $\endgroup$ – Glen_b -Reinstate Monica Nov 18 '16 at 11:49
  • $\begingroup$ @Glen_b I know but I'm trying to find a way to describe these effects and put them in my data, however I'm still searching. Did you know a good strategy? $\endgroup$ – Francesco Nov 18 '16 at 16:13

I would begin by characterizing the observed data with a time series model (arima + identified determistic dummy indicators ) that reasonably separate signal from noise. With that model ( and estimated parameters ) I would then inject random errors to get a realization/synthetic data set. I would do this N times to get a family of possible realizations. One of the issues that should be addressed as you construct your useful model is to incorporate identifiable deterministic structure such as pulses/level shifts/time trends/seasonal pulses . Clearly pulses can arise at any time so your procedure/script should encode such possibilities while level shifts/time/trends/seasonal pulses would be invariant/fixed in your synthetic data.

I am currently delivering/researching simulated forecasting providing a family of forecasts for each period in the future based upon a model and a set of parameters see AR(1) forecasting. Your very interesting question may motivate me to routinely deliver the N traces that you are after as they become actionable input to a decision-theoretic post processor.

The "fact" that you can characterize the observed set as normal is insufficient as the observations are auto-correlated thus simulation needs to incorporate this characteristic.

  • $\begingroup$ What about trying a completely different strategy? My objective is find a way to describe this distribution so that I can generate in future similar trend but with custom statistics (like mean, sd, etc) and later try to generate a full dataset of different distributions related also in a non-linear way. So I was thinking: trying something like rejection sampling could be useful to my research? $\endgroup$ – Francesco Nov 18 '16 at 16:15
  • $\begingroup$ I am not sure why rejection sampling would be necessary. Can you explain why you think you may need it ? $\endgroup$ – IrishStat Nov 26 '16 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.