1
$\begingroup$

I'm still playing with the data related to the year 2008 of the "Household power consumption" dataset (free to download at UCI Machine Learning Repository). I was able to generate some synthetic data but now I have a new question: how can I evaluate the quality of my synthetic data? Considering this distribution as the ground truth:

Original data

How can I (for example) find the best between these other two distributions?

Synthetic data 1

Synthetic data 2

At the beginning I thought that I could base the quality on the similarity / distance between my synthetic data and the generator but now I don't think that it's enough, because I want to create something that is also a little bit different (or far if we talk in distance terms) in trend but is good to be used as synthetic data.

Improve this question
$\endgroup$
3
  • $\begingroup$ This completely depends on what you want to use it for. Depending on the application, an exact copy of the original data or replacing all datapoints by the average for example might be good candidates or horrible options. $\endgroup$
    – dimpol
    Commented Nov 30, 2016 at 10:40
  • $\begingroup$ @dimpol I don't want an exact copy, I want to generate datasets that are at the same time similar to the original data but have their characterstics, for example, have peaks in different points or could show more/less noise. My objective is to find a way to say that synthetic dataset A is better/worse than synthetic dataset B. $\endgroup$
    – Francesco
    Commented Nov 30, 2016 at 10:54
  • 2
    $\begingroup$ There are different similarity measures that measure different things. Which one you need completely depends on what you want to do with it. It is like coming to us with an apple and saying: "I want things similar to this". It could be more apples, more fruit, more groceries, more red products, more products of that size/weight, more products of that price, etc. All are 'similar' in a way, which one you need is something only you know. $\endgroup$
    – dimpol
    Commented Nov 30, 2016 at 11:00

1 Answer 1

Reset to default
1
$\begingroup$

As @dimpol notes, this will heavily depend on what you want to use your similarity for.

One way forward would be analogous to so-called "visual inference": generate a synthetic dataset according to your rules. Show relevant plots (like your time series plots, or whatever kind of plot is appropriate to your question, like a scatterplot) of your synthetic dataset along with the same plot for "real" data to a number of people and ask them to identify the synthetic dataset. If only few people (i.e., less-than-by-chance-alone) can identify your synthetic among the real data, then your generation rules are good enough in your specific domain.

The disadvantage is of course that you need a number of human participants. You may be able to simulate the entire procedure using automatic tools instead, using some kind of classifier. If your classifier cannot reliably distinguish your synthetic from true data, your rules are good. (Or your classifier isn't.)

On visual inference, see Buja et al., 2009, Philosophical Transactions: Mathematical, Physical and Engineering Sciences. On the "automatic" idea, you may want to take a look at Generative Adversarial Networks, whose idea is similar to what I outlined above.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.