Let's say that I have two datasets. The target dataset is 1K instances, the "predicted" dataset is 1M instances (but I could downsample).

I can compute the distance between instances.

How do I quantify how much the two datasets are "similar"? e.g. drawn from the same distribution?

As an example, if the predicted dataset oversamples a particular instance (or near a particular instance) in the target dataset, that would be bad.

Additionally, if the predicted dataset has instances that are very far away from all the target instances, that would also be bad.

What are appropriate well-motivated measures for this task? Particularly with good Python implementations, but not required. I have looked for similar questions, but most seem to be based around comparing individual samples to a particular dataset or 1d datasets.

I am aware of the Ben David Discepancy approach but would want to use distances (between target and predicted dataset instances) to solve my problem.

  • $\begingroup$ Do you have the points $x_i,y_j$ from the two datasets, or just the distances $d_{ij} = d(x_i, x_j)$ ? $\endgroup$ – ArnoV Mar 5 at 8:51
  • $\begingroup$ This answer might be helpful to you, if you need more information please ask a more specific question. $\endgroup$ – ArnoV Mar 5 at 8:53
  • $\begingroup$ ArnoV: I have the points from the two datasets. They are audio, so each point is 100K dimensional. There are some semi-reasonable distance measures for audio, based upon multi-scale spectrum distance $\endgroup$ – Joseph Turian Mar 5 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.