Synthetic data behaviour different from real data

Question

I'm studying in R a way to generate synthetic data starting from real world data and, in particular, I'm focusing on the year 2008 of the "Household power consumption" dataset (free to download at UCI Machine Learning Repository).

My problem is that the synthetic data are far from real data and I'm can't understand why.

These are the steps that my process has so far:

1. I imported the data for the year 2008 and I aggregate them on hourly basis.
2. I'm started to focus on the less related variable in the dataset (voltage) and I find all the components using the command decompose(). I removed the noise effect and seasonality (I will add them later).
3. I divided the data in different seasons and I started from the winter, checking the histogram and I found that a normal distribution could be a good fit form my data:

4. So I generated my data using fit_w <- rnorm(length(w), mean(w, na.rm = T), sd = sd(w, na.rm = T)), but the final results are not so satisfying:

My question is: how can I generate data in a way that they will look similar to the real dataset? Because all my efforts and strategies that I tried until now will lead to similar results.

Answer 1

I would begin by characterizing the observed data with a time series model (arima + identified determistic dummy indicators ) that reasonably separate signal from noise. With that model ( and estimated parameters ) I would then inject random errors to get a realization/synthetic data set. I would do this N times to get a family of possible realizations. One of the issues that should be addressed as you construct your useful model is to incorporate identifiable deterministic structure such as pulses/level shifts/time trends/seasonal pulses . Clearly pulses can arise at any time so your procedure/script should encode such possibilities while level shifts/time/trends/seasonal pulses would be invariant/fixed in your synthetic data.

I am currently delivering/researching simulated forecasting providing a family of forecasts for each period in the future based upon a model and a set of parameters see AR(1) forecasting. Your very interesting question may motivate me to routinely deliver the N traces that you are after as they become actionable input to a decision-theoretic post processor.

The "fact" that you can characterize the observed set as normal is insufficient as the observations are auto-correlated thus simulation needs to incorporate this characteristic.