Mean of means -> error propagation or uncertainty or both?

Question

my problem is as follows. I have a simulation of a neural network which creates activity patterns, learns them and then tries to retrieve previously learned patterns one by one. The performance of these retrievals (=remembering) is the result I'm interested in.

Simplified, this is how it works (I used fictive numbers to make it more understandable):

• Create 40 random activity patterns within a network of 200 neurons.
• Learn those patterns.
• Retrieve (=remember) each pattern. This gives me a retrieval performance (between 0 = "not at all" and 1 = "perfectly"). Now, this retrieval process uses randomly generated numbers and will be performed 40 times for each pattern leading to a mean performance for the given pattern set. I also calculate sample standard deviation and uncertainty of the mean.

My problem is now this: I change some parameters, say the number of patterns created in the first step, and for each number of patterns, I run the whole thing (create patterns, learn them, retrieve them) 40 times. Then, I have for each number of patterns 40 mean performance values plus their corresponding standard deviations and uncertainties of the mean.

I know want a single mean (of all these 40 performances) for each number of patterns.

That's easy, but how do I calculate the uncertainty of this overall performance value? Should I use gaussian error propagation for the uncertainties of the mean values I'm averaging over? Or should I just use the uncertainty of the newly calculated overall mean performance??

So, this is how it looks like:

# of patterns    (mean performance)1    (mean performance)2    ...
10               0.82 +- 0.01            0.79 +- 0.05          ...
11               0.75 +- 0.02            0.77 +- 0.03          ...
12               ...                     ...
...

I calculate the mean for all (mean performance) values corresponding to a number of patterns, so basically I'm taking the mean of mean values.

If you could give me a hint of how to best calculate the overall uncertainty of the result, I'd very much appreciate it.

I've googled a lot and also read some information sheets about errors in measurements but am not sure as how to best proceed.

My current idea is to just use gaussian error propagation but then I won't take the uncertainty of the newly calculated mean value into account.

Answer 1

This is a job for "bootstrap resampling". Use uniform random sampling with replacement to compute the mean some good number of times (hundreds, tens of thousands, depends on your data). I suggest not more than 5x the number of values used to compute the mean of means but it is heuristic. Compute the resampled mean for all of those subsets. The variation in that resampled-mean list will give you an estimate on the variation in the mean of means. I would use inter-quartile range as a pseudo-sigma (divide by 1.35 to get a robust estimate of standard deviation).

HEre are some links

• http://physics.ucsc.edu/~peter/jackboot.pdf
• http://courses.washington.edu/matlab1/Bootstrap_examples.html
• http://wise.cgu.edu/downloads/Introduction%20to%20Resampling%20Techniques%20110901.pdf