Building a bootstrapped distribution to test reliability of empirical data I have data from a study where subjects (considered as "generations" in a Telephone-like game) sang short melodies from one to the next, starting from a randomly-generated melody (seed) given to the subject who acted as the first generation.
The dependent variable was the average number of unique pitches (specifically, pitch classes, "PCs") across generations. The data came from 5 different experiments differing only in parameters not relevant here. Relevant for the conclusions of the study is whether a decrease in this number took place across generations, and if so, for which experiment version.
Plotting the data shows a decrease in two of the expt. versions (3f and 3m):

However, it seems frugal to just claim this based on the plot, without any sort of statistical test. I don't refer here to effect size (whether there's been "enough" of a decrease), but to how likely it is for such an effect (of any size) to have arisen through chance alone.
I'm guessing bootstrapping would be the specific "control" against which to test for the significance of said decrease in the number of unique picthes. But it is unclear to me how the null/bootstrapped distribution should be obtained in this case, and how exactly I would then go ahead with the significance test (I've never used bootstrapping, just know of it).
In the plot above, I included the DV's value for the seed, as that is as good a control as any, for x=1. However, I guess the bootstrapping would have to extend across the same number of data points (8 generations across the x-axis, in this case), and not just cover me for the first time point.
I did these analyses in Matlab, if relevant. Any thoughts how this test could be implemented (or what extra details I need to give to make the question answerable) would be very much appreciated.
 A: This sounds more like a candidate for a permutation test.

Relevant for the conclusions of the study is whether a decrease in this number took place across generations.

So your hypothesis is that there would be a decrease in PCs with regard to generations. This means that your null hypothesis is that there is no such decrease, or that all the generations are roughly the same, or "random". In such a case, the permutation test would randomly permute the generation numbers to simulate a null distribution where PCs of different lengths can happen in any generation, regardless of order.
The next decision that you need to make is how do you operationalize "decrease". There are tests for trends, but you want to use a permutation test, so you need to define some statistic that measures the decrease (e.g. the difference between the first and last generation, the average difference between subsequent pairs, the slope of linear regression, etc). In the permutation test, you would calculate the statistic over the "null" data and then compare the result from experimental data to the distribution of the statistics estimated over the null re-samples. This would tell you how likely the statistic would be if the null hypothesis was true, the $p$-value.
I don't use Matlab, so let me use Julia to give an example, as the syntax is quite similar.
using GLM
using Random
using Statistics

generation = [0:8;]
# Expt 3m data scrapped from your plot 
unique_PCs = [7.6, 7.7, 7.1, 6.6, 6.7, 6.4, 6.6, 6.75, 6.2]

X = hcat(ones(length(generation)), generation)
y = unique_PCs

# linear regression slope
slope = coef(lm(X, y))[2]

null_slopes = []
for i in 1:5000
    # randomly permute the rows of X
    X_null = shuffle(X)
    null_slope = coef(lm(X_null, y))[2]
    append!(null_slopes, null_slope)
end

# calculate the probability that the observed slope
# is lower or equal to the slopes from the null distribution
mean(null_slopes .<= slope)
# 0.0056

As you can see from the example above, the probability of observing as small regression slope as for Expt 3m in the "random" data under null distribution is low.
