Statistically compare two large continuous datasets

Question

I have multiple images with 1 band/channel. This of an RGB image, then I only have the blue band/channel. In other words, multiple 1D datasets, or multiple 1D arrays.

I would like to statistically compare each pair of images where a pair means two successive images.

Each image contains about 50,000 pixels or values. This means that one can have 50,345 values and the other 50,433 values, so the number of values is not dramatically different but it is not always the same, so any method that is based on equal arrays will not be adequate here. This also means that the pixel in coordinates, x,y in image (or array) #1 does not have to correspond to the pixel in the same location in image #2.

Let's take these two examples (where each color corresponds to different image or array of values):

Put in a non-statistically way, the blue and the red are similar, while the red and the green are different.

I would like to perform a statistical test that will quantify this difference and then I can choose a threshold and decide accordingly if these are similar enough for my application or not.

My question is - which statistical test or model or method is adequate for that assuming the distribution is similar to what you see in the examples, meaning that the distribution is not 100% Gaussian.

The t-test and z-test do not work here because the degree of freedom is huge hence the p-value is 0, see for example one (of many) t-tests variations I did:

stats.ttest_rel(img1,img2,nan_policy='omit')
>>> Ttest_relResult(statistic=-90.27773456178737, pvalue=0.0)

stats.ttest_ind(img1,img3,nan_policy='omit',equal_var=False)
>>> Ttest_indResult(statistic=360.2704559875767, pvalue=0.0)

I thought maybe to try to calculate the distance between the datasets or to calculate the overlapping histogram area between two datasets (because it seems better than comparing the mean) but I'm not sure which method (preferably in Python) is adequate for such a task.

At the moment, I can't quantify or define "similarity" for my application. I will be able to do that once I have a number that will quantify the similarity and then I'll have to check more examples and see which threshold is ok for me. So I do not need an answer to the similar/not similar question, rather I would like to get an answer as to how to quantify this similarity. My final goal (which is not the question here) - is to get a true/false result. i.e., are these datasets similar (true) or not (false), based on a value that will quantify the similarity (this is my question).

I know my question is a bit like shooting in the dark but this is because I am not sure which way to go- should I compare the means? the variance? the area of the histograms.

One last thing: I would like to be able to automate the solution since I have many of these paired datasets, so visual inspection will not work here.

Answer 1

You could try some sampling techniques. In more detail, you could select smaller random samples from the blue, red and green populations and compare those samples using the traditional statistical tests you mentioned. Run that multiple times and count how many times the null hypothesis (that the means are equal) got rejected out of the total. Keep in mind that p-values are random variables too so at a significance level of 5%, you'd expect 5% of these hypotheses tests to reject the null hypothesis even when the means are the same (so potentially even in the red vs blue case).

Alternatively, another option would be to run the Kolmogorov-Smirnov test.

Answer 2

One feature (not a bug) of hypothesis testing is that it gets more sensitive to small differences as the sample size increases. Consequently, hypothesis testing considers more than just effect size, and you’re really only interested in the effect size (perhaps in addition to some quantification of the uncertainty).

However, the description of your problem suggests that you will always have a sample size of the $50000$ pixels in your image. I suspect those pixels are not independent of one another (if a picture of a black dog has a black pixel, I say there’s a good chance that nearby pixels will also be black), but maybe you’re willing to make such an assumption; let’s assume so. In such a case, differences in the p-value will be due to differences in effect size and nothing more, so the p-value will be a decent measure of distribution similarity.

To handle the p-value being tiny, you might consider taking a logarithm and determining your threshold on the log scale.

However, you would be doing this to get at the effect size, so I would suggest looking directly at the effect size. You can use your software to calculate the difference in means along with confidence intervals, using those to make your decision. Perhaps even better would be to use the approach from the Kolmogorov-Smirnov test and find the maximum vertical distance between the empirical CDFs (along with a confidence interval for such a value), which will be sensitive to differences other than the mean. Another option to which you allude when you mention the overlap of the histograms is the Earth-mover’s distance. Yet another option is KL divergence.

(Note that a such an approach using confidence intervals still relies on independence of the pixels, which I doubt.)

Answer 3

With such large sample sizes, you may get a clearer view of the relatively small (but obvious) differences between red and blue by looking at your histograms than from formal tests.

Consider the (roughly similar) fictitious data below:

set.seed(2022)
r = rbeta(50000,7,3)
summary(r)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.09307 0.60781 0.71252 0.69911 0.80363 0.99693 

b = rbeta(50000,7.5,2.5)
summary(b)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.1610  0.6658  0.7676  0.7495  0.8500  0.9974 

Plotting kernel density estimators:

plot(density(b), col="blue", lwd=2, 
 ylab="Density", xlab="value", main="KDEs")
lines(density(r), col="red", lwd=2)

Both distributions have support $(0,1)$ and there is some skewness, so there is some doubt whether t tests are precisely accurate, even if they do show a highly significant difference.

t.test(r, b)

    Welch Two Sample t-test

data:  r and b
t = -59.146, df = 99710, p-value < 2.2e-16
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -0.05205704 -0.04871757
sample estimates:
 mean of x mean of y 
 0.6991094 0.7494967 

Because of slightly different shapes and dispersions, a Wilcoxon rank sum test shows stochastic domination of blue over red (rather than a just a difference in medians).

wilcox.test(r, b)

    Wilcoxon rank sum test 
    with continuity correction

data:  r and b
W = 983310000, p-value < 2.2e-16
alternative hypothesis: 
 true location shift is not equal to 0

It seems more direct to look at your histograms of the actual data where results are obvious than to make excuses for tests that may not be exactly applicable.

Answer 4

The t-test and z-test do not work here because the degree of freedom is huge hence the p-value is 0

Yes. If you're testing the null hypothesis that the two datasets are unrelated, the p-value will be tiny. Even for the red versus green datasets, you should reject the null hypothesis.

If you're not testing the null hypothesis that the two are unrelated, but merely trying to quantify how much they differ, there's many different metrics. Your description isn't really clear on what exactly you want to test, but if you want to ask "are these curves the same", you'll want a vector norm. One main type is the $L^p$ norm. In this norm, for a particular $p$ value (note: this is completely different from the p-value in the sense of probability) you take the sum over all $x$-values of $|f(x_i)-g(x_i)|^p$, and then take the $p$-th root of that value. This yields a different norm for each $p$, which can range from $0$ to $\infty$ ($L^{\infty}$ norm is just the maximum). $L^2$ is the Euclidean/Pythagorean norm.

There's also the covariance. Both of $L^p$ and covariance are dependent on the scale of the distribution. That is, doubling both $f$ and $g$ will result in larger values. If you don't want that, you can normalize them. If you divide the covariance by the product of the the standard deviations, you get the correlation.

Answer 5

I'm not sure what your data represents, but in landscape/spatial ecology, it's common to have multiple raster datasets representing different variables for a given spatial area. One of the first issues that comes up is Spatial Autocorrelation.

Put simply, spatial autocorrelation occurs when you have measured variables at two points close enough together in space that they are not independent, which in turn can undermine the assumptions of your statistical tests (such as your t-test). So the first thing you have to do is figure out if spatial autocorrelation is an issue for your data. I'm not an expert on this, just familiar with the issue, so you'll have to spend some time researching the methods for this. One measure that you will definitely come across is Moran's I, which is probably a good place to start.

If you determine that spatial autocorrelation is an issue, then one way you can deal with it is by subsetting your data points so that they are far enough apart that they can be considered independent (there may be other ways that I'm not familiar with). There are statistical tools for determining how far is necessary, but it's been so long since the one time I had to do it that I can't remember what they are. Especially because my dataset at the time was too big and I couldn't get them to run, so I ended up picking a distance that I could justify biologically based on my knowledge of the study system.

Answer 6

In machine learning for generative adversarial networks (GANs), it is common to use Wasserstein metric / Earthmover's distance to compare output images from the well-known paper 2017 on Wasserstein GAN. Traditional measures like K-L divergence may be too stringent: see What is the advantages of Wasserstein metric compared to Kullback-Leibler divergence?

Also if your goal is to compare specifically images for being nearly visual duplicates, you can look into image perceptual hashing/fingerprinting: pHash, Neal Krawetz's classic introductory blogpost on perceptual hashing (can be as simple as squashing image down to 8x8 and binarizing, then comparing by Hamming distance)

Answer 7

I would like to perform a statistical test that will quantify this difference and then I can choose a threshold and decide accordingly if these are similar enough for my application or not.

A statistical test will be based on statistical variations and will rank samples based on probability.

For instance the chi-squared test and Kolmogornov-Smirnov will assume a particular distribution or particular assumptions about the distribution and compute a statistic with those assumptions.

You might want to look to other measures of distance instead. Measures that are not inspired by statistical hypothesis testing.

My question is - which statistical test or model or method is adequate for that assuming the distribution is similar to what you see in the examples

Those two images are not enough to create a model of the data and come up with an appropriate distance measure for which you can select a cut-off value.

To come up with a model requires an understanding of the process that generates the data. You can not just look at two examples of data output and decide what would be a good model from which a distance measure can be defined. (However if you have thousands of examples, then you could use some neural network to come up with a model learned from the examples).

Answer 8

Since you are dealing with histograms of the channels of an image, you could consider using OpenCV to compare the histograms via a distance metric to express how well they match.

In OpenCV, this task is somewhat trivial: you could use the function cv.compareHist to compare histograms using a given metric; you can select among four different distance metrics: Correlation, Chi-Square, Intersection, and Bhattacharyya.

A handy tutorial on histogram comparison using OpenCV is here.