I asked this question over on Physics.SE to get some practical experience from experimentalists on how to express uncertainty or confidence in data when one can only afford a single run of the experiment. I'm hoping to get some additional information from the broader statistics community on how to tackle the problem at hand.
To recap and generalize slightly, I have a highly-nonlinear (physical) system with many inputs and many outputs, and I can only afford to run the full system once. It can be an experiment or a simulation -- the state-space in either case is going to have a very large dimension, so large that it's effectively infinite (although technically they are finite-dimension I suppose). Because it is so expensive, I will extract out as much data as I can with a variety of measurement techniques and processing. There will be data over both time and space, as well as global and pointwise quantities. I'm going to throw the kitchen sink at it!
Since the system is too expensive to run more than once, I can't repeat data points (same inputs, compare outputs to assess uncertainty) nor can I vary inputs to assess variations in the outputs. So I cannot build a covariance matrix for the full system, using the full system.
I do have some insight into what kinds of processes might happen though, and I can design an unlimited number of canonical/simplified tests that can mimic those processes in isolation or in representative combinations. Of course, even if I represent all of the right combinations of processes, they won't be in the full system so they are still only approximate. Some examples of what I mean are further down in the question.
How can I use these simplified tests to express something meaningful about the final dataset from the full experiment?
I know "uncertainty" or "confidence" have specific statistical meanings and I don't want to limit the question to just those, but in lay-terms, I'd like to be able to express uncertainty or confidence (or lack thereof) in my final measurements so I can make decisions about what they mean.
I'm looking for a rigorous statistical framework within which I can do this. The only fruitful direction I can think of is to use the simpler tests to somehow cook up an approximation of the covariance matrix, but I feel like that is pushing the "engineering-judgement" (a.k.a. intuition and guessing) from interpreting the final results to cooking up some priors, which is harder to justify to other engineers. I also feel like that cooked-up covariance matrix would be very sparse relative to the real one but perhaps there is a way to figure out if that's the case or not.
Some examples of what I mean by canonical/simple tests that are representative of the full system:
Imagine I am sending an interplanetary probe that will descend through the atmosphere of a gas giant until it gets destroyed. I only get one shot at this, so I will only be able to collect one dataset. Prior to leaving Earth, I can test my various sensors on simple mixtures of gases I expect it to encounter, at a few pressures and temperatures I can recreate in my lab. From this, I can build up some information about how the reported values vary. But, I can't hit the full complexity of states that the probe will encounter in the real atmosphere. The gas mixtures might be far more complex, or the pressures and temperatures are very different, or the sensors might be in a position where the shock from the probe moving causes chemical reactions near the sensor and so its seeing things we never thought to test before it left.
I will fire up a simulation of the entire ocean and it's going to take about a year of continuous running to finish. So, I'm only going to get one chance at doing it and I need to understand how my physical models and my grid, time step, etc. are going to impact the results. I can run many simpler experiments on isolated physical models and on smaller grids -- for instance, I can run some mixing studies to see how my salinity model behaves. And I can run some small problems with different heating from the sun to see how that impacts heating. Or I can run small problems to see how my pollution chemistry model behaves. But, in the full system, these things will interact non-linearly but I'd like to take what I know from the simpler cases to know how to interpret the full results.