How can I tell if images are from the same distribution? What are some tests I could run on two groups of imagery to see if they're from the same distribution? Imagine, for example, that you want to augment a dataset with another, and you want to make sure they're not too dissimilar first. I understand that there won't be a single test that will guarantee this so I'm looking for a few different tests that I could run to make sure that the groups of imagery aren't too different.  One option is to make a histogram of all the color channels and compare them (either statistically or visually). What are some other options?
 A: If you don't have too many ties in the data you can use ks.test in R.
This is R's implementation of several kinds of Kolmogorov-Smirnov tests.
The one you want has two data vectors, say x and y, from two independent samples.
The null hypothesis is that they come from the same distribution.
Examples:
set.seed(2020)
x = rnorm(30, 100, 15)
y = rnorm(50, 100, 15)
z = runif(30, 50, 150)

No rejection because x and y are from the same distribution, even
though sample sizes differ:
ks.test(x, y)

        Two-sample Kolmogorov-Smirnov test

data:  x and y
D = 0.22, p-value = 0.2864
alternative hypothesis: two-sided

Rejection because y and z are from different distributions.
ks.test(y, z)

        Two-sample Kolmogorov-Smirnov test

data:  y and z
D = 0.32, p-value = 0.03467
alternative hypothesis: two-sided

Samples from different distributions have to be moderately
large for rejection. However, there is a cap of several
thousand on the largest sample size. See the documentation.
The test statistic $D$ for this two-sample K-S test is the
maximum vertical discrepancy between the empirical cumululative
distribution functions of the two samples. In the graph below, the ECDFs for
data vectors x, y, and z are plotted using colors
maroon, green, and blue, resptectively.

par(mfrow=c(1,2))
 plot(ecdf(x), col="maroon", main="x vs. y")
  lines(ecdf(y), col="darkgreen")
 plot(ecdf(y), xlim=c(50,150), col="darkgreen",
      main="y vs. z")
  lines(ecdf(z), col="blue")
par(mfrow=c(1,1))

A: Fréchet Inception Distance (FID) is a widely accepted standard for evaluating the distance between image distributions. The exact details are available from the original paper, but to make a long story short, you pass all the images through a pretrained neural network (usually a standard "Inception" network, hence the name) and they're all encoded into some $k$-dimensional vector. Then you fit two separate $k$-dimensional gaussian distributions to each set of images accordingly, and measure their Fréchet distance, which, for gaussian distributions, has a closed form $||\mu_0 - \mu_1||_2^2 + \text{tr}(\Sigma_0 + \Sigma_1 - 2(\Sigma_0\Sigma_1)^{1/2})$.
