Parameter estimation from single dynamic experiment. How many degrees of freedom?

Question

I am estimating a parameter (diffusivity) from a set of experimental data coming from the same dynamic experiment: it can be summarised as changing the oxygen partial pressure around a solid and measuring the mass uptake with time (it is actually somewhat more complex than this but for the sake of this question, this description suffices). This experiment was carried out only once and, as it is measured online, it provided a lot of data. I estimate the diffusivity of oxygen for trying to assess whether the diffusivity changes when changing the relative humidity.

I did a first estimation (least-square minimisation) using 8 data and I obtained the following figure

The estimated value of the diffusivity is $D = 0.10$ and a $ \alpha = 0.05$ confidence interval would be approximated by $[0.06 - 0.17]$. To obtain the confidence interval I used the asymptotic approximation $$ \hat{\theta} \pm t^{\alpha/2}_{n-m} \hat{\sigma} \sqrt{diag(J^{T}J)^{-1}} \tag{1}$$ where $n$ is the number of data, $m$ is the number of parameters (in this case 1), $J$ is the Jacobian of the regression function and the unbiased estimator of the error variance $$\hat{\sigma}^2 =\frac{\sum_{i=1}^n (y_i-f(\hat{\theta}))^2}{n-m} \tag{2}$$

If I repeat the estimation of the parameter using 62 data points I obtain

Now, the estimated value of the diffusivity is $D = 0.130$ and the confidence interval $[0.122 - 0.138]$. What troubles me here is that, these additional data points are not "independent" in the sense that, coming from the same experiment, we can expect them to be dependent on the previous data point. Likewise, the second figure does provide more information to estimate the parameter but, it would seem that if the measurements can be taken online, the parameter (in this case, the diffusivity) can be estimated with arbitrary confidence, which of course, is not true.

My intuition is that those extra data points provide some information that allows a more accurate estimation. But only to a certain extent. To be really considered as independent data, each should come from an individual experiment (which in this case would be unfeasible).

So, here comes my question: How many degrees of freedom are there in a single dynamic experiment such as this one? And how should the different number of data be used when comparing the obtained parameter (e.g. t-test) with another one at different conditions?

Answer 1

Statistical independence/dependence does not make reference or concern itself with whether or not the data comes from the same experiment but about the data generating process. Independence in statistics is whether or not the data points influence each other, not whether or not they come from the same experiment. So your concern of "What troubles me here is that, these additional data points are not "independent" in the sense that, coming from the same experiment, we can expect them to be dependent on the previous data point." is not a concern in general but likely because of your specific experimental setup.

That is it's a different concern to say "my data are all from one experiment and so I might have a problem" than to say "my data are not independent". In your case you have both problems. However, the only way to address the limitation of "my data is just from one experiment" is to run more experiments. However then you get into the issue of "all my data is just from one experimental apparatus" so you need to find another experimental apparatus... so on and so forth. Another way to address this is to say that the all from one experiment concern addresses "systematic" error and the "data are not independent" concern reflects the concern that your uncertainties are miscalculated from that experiment (i.e., you are misestimating the "random" error.

Furthermore, for these models and this setup that you are doing I would argue that "degrees of freedom" is an ill defined concept and so you just shouldn't worry about it. The trickiness with defining it for your particular experiment is that it isn't clear in how many ways you can vary/change your system independently. I guess my answer is: you're asking a good question and the answer is unfortunately that the concept just doesn't really apply.

You are doing diffusion experiments and I have some familiarity with them though likely in a very different context and it sounds to me like you are measuring how quickly oxygen diffuses into a material (if this is wrong then your model will need to be more complicated). Let's go with measuring oxygen uptake in a material to measure the diffusion coefficient. So there are some possible parameters that your model is interested in fitting: the diffusion coefficient and the measurement error.

Instead of fitting a curve to your data as you have done now which leads you to all sorts of nasty considerations such as what happens when the data are not statistically independent (that is the data depends on other data points; in your case points depend on the value before them). I would propose you setup a physical model of your system and then fit the parameters using a bayesian model. The basic idea I am proposing is as follows:

1) write a physical model that generates the data from a diffusion coefficient 2) use a MCMC sampler to fit the diffusion coefficient and measurement error that best fits your data along with the uncertainties

For the MCMC sampler you will need to use a likelihood function and for that I propose weighted square deviations:

$$L = \sum_i {(y_i-s_i) \over \sigma^2}$$

where $$s_i$$ are the simulated data that correspond to each $$y_i$$ and $$\sigma^2$$ is the measurement error that you are fitting from your data.

Assuming that the MCMC sampler does it's job and finishes you should get both the parameter values and uncertainties without having to explicitly worry about degrees of freedom.

Here is some example code for a similar situation: https://github.com/bblais/pyndamics

Answer 2

What you're dealing with here is a common task in the field of system identification: You're trying to estimate one parameter $D$ (diffusivity) of your model $C(D, t)$ (which is hypothesized to be the true model, normally found by applying first principles) based on measured data $C_m(D, t)$:

$C_m(D, t) = C(D, t) + n(t)$

Where $n(t)$ is measurement noise. $n(t)$ should be independent of $C(D, t)$. Furthermore, very often it is assumed that $n(t)$ is white noise with a constant variance $\sigma$. Under these assumptions, all $N$ samples of $n(t)$ are iid and you can construct confidence intervals for your estimation of $D$ with $N-1$ degrees of freedom.

So the main point is that your intuition is correct: The $N$ samples of $C_m(D, t)$ are not idd due to the function $C(D, t)$ being deterministic, but the $N$ samples of measurement noise $n(t)$ should be iid.

More details can be found in books about System identification. The classic reference is: Lennart Ljung: System Identification — Theory For the User, 2nd ed, PTR Prentice Hall, Upper Saddle River, N.J., 1999.

Answer 3

Suppose that your instrument measures $$y=f(D)+n$$ where $f(D)$ is the exponential diffusion curve that you plotted and $n$ is the measurement noise. Since you are talking about a single experiment we assume that $f(D)$ is deterministic and only the measurement noise is random. When you are talking about independence and degree-of-freedom, it concerns $n$, not $D$.

You wrote "the evolution of a deterministic system depends on the previous state. Therefore, the concentration at t+1 will depend on the concentration at t. Eventually, all the data points depend on the initial one."

This does not make any problem. This actually enables us to use all the points in the experiment to estimate one single parameter $D$, assuming a model function $f$.

In the case of a single experiment assuming that all the uncontrolled conditions are at a single value, the confidence interval only concerns the measurement process. Every measurement equipment has a bandwidth: it can produce a limited number of independent observations in a time span. Hopefully, the bandwidth of your measurement equipment is large enough such that it can measure $f(D)$ without any distortion. But we like to make as many observations as possible to improve the accuracy of the estimation.

You need to answer this question: "Do you already see bandwidth limitation in $n$?

You can calculate the residual of your estimate and study their autocorrelation function or their power spectral density. In the case that you see a strong correlation, you can calculate an equivalent number of independent samples and adjust your estimate of the variance of parameters. For example, is your collect 100 samples per second for a duration of 1 second then you have 100 samples. If you analyze the residuals and see that the bandwidth of the data is 25Hz instead of 50 Hz it means that half the samples are effectively redundant and then you need to multiply your variance by 2. Or you can downsample your signal to remove the correlation. If you don't see any correlation in the samples then the degree of freedom would be the number of samples minus one. But the confidence interval concerns the measurement process not the diffusion constant.

The fact that confidence interval is reported from a single experiment, however, is troublesome in my view, unless the propose of the experiment is to demonstrate the accuracy of the measurement apparatus, not the diffusion constant. We report the number because it supposed to be reproducible. So at least given all your conditions and instruments if someone repeats the experiment then she should be able to get similar numbers and we report confidence interval to make possible to measure similarity.

Also a side note About selecting the point: When estimating the parameter in a decaying process, e.g.: $$y = A - B \exp(-Dt) + n$$ The points close to $t=0$ are the most important ones. Since as $t$ increase the signal component to noise ratio becomes smaller.