1
$\begingroup$

I am estimating a parameter (diffusivity) from a set of experimental data coming from the same dynamic experiment: it can be summarised as changing the oxygen partial pressure around a solid and measuring the mass uptake with time (it is actually somewhat more complex than this but for the sake of this question, this description suffices). This experiment was carried out only once and, as it is measured online, it provided a lot of data. I estimate the diffusivity of oxygen for trying to assess whether the diffusivity changes when changing the relative humidity.

I did a first estimation (least-square minimisation) using 8 data and I obtained the following figure

enter image description here

The estimated value of the diffusivity is $D = 0.10$ and a $ \alpha = 0.05$ confidence interval would be approximated by $[0.06 - 0.17]$. To obtain the confidence interval I used the asymptotic approximation $$ \hat{\theta} \pm t^{\alpha/2}_{n-m} \hat{\sigma} \sqrt{diag(J^{T}J)^{-1}} \tag{1}$$ where $n$ is the number of data, $m$ is the number of parameters (in this case 1), $J$ is the Jacobian of the regression function and the unbiased estimator of the error variance $$\hat{\sigma}^2 =\frac{\sum_{i=1}^n (y_i-f(\hat{\theta}))^2}{n-m} \tag{2}$$

If I repeat the estimation of the parameter using 62 data points I obtain

enter image description here Now, the estimated value of the diffusivity is $D = 0.130$ and the confidence interval $[0.122 - 0.138]$. What troubles me here is that, these additional data points are not "independent" in the sense that, coming from the same experiment, we can expect them to be dependent on the previous data point. Likewise, the second figure does provide more information to estimate the parameter but, it would seem that if the measurements can be taken online, the parameter (in this case, the diffusivity) can be estimated with arbitrary confidence, which of course, is not true.

My intuition is that those extra data points provide some information that allows a more accurate estimation. But only to a certain extent. To be really considered as independent data, each should come from an individual experiment (which in this case would be unfeasible).

So, here comes my question: How many degrees of freedom are there in a single dynamic experiment such as this one? And how should the different number of data be used when comparing the obtained parameter (e.g. t-test) with another one at different conditions?

$\endgroup$
  • $\begingroup$ Welcome to the site. You wrote "What troubles me here is that, these additional data points are not "independent" in the sense that, coming from the same experiment, we can expect them to be dependent on the previous data point. " Can you say more about why you think this? Why should they be dependent on the prvious data? $\endgroup$ – Peter Flom - Reinstate Monica Sep 24 at 10:54
  • $\begingroup$ Well, the evolution of a deterministic system depends on the previous state. Therefore, the concentration at t+1 will depend on the concentration at t. Eventually, all the data points depend on the initial one. I would consider them as independent if I reconstructed the same profile (62 data points) with data from 62 experiments and, from each experiment, I would take only one data point at different sampling times. Again, it is more my intuition than something I can formalise $\endgroup$ – Toulousain Sep 24 at 11:02
  • $\begingroup$ What is the origin of this confidence interval? It strangely admixes two nearly contradictory theoretical elements: the asymptotic maximum likelihood theory and the small-sample Student t distribution. Perhaps your question could be resolved simply by using a different confidence interval procedure. $\endgroup$ – whuber Sep 24 at 11:28
  • $\begingroup$ I have to admit I don't follow exactly your point @whuber. Is it about equation (1), which I have used until now repeatedly? Or my comment "e.g. (t-test)"? Which different procedure would you suggest? $\endgroup$ – Toulousain Sep 25 at 11:42
  • $\begingroup$ Because the fits aren't good, I would suggest (a) using a different model that (b) allows for correlation among the errors and (c) uses standard ML confidence intervals (which typically are based on the standard Normal distribution rather than the t distribution and use the MLE of $\sigma$ for its estimate). Also, because it's likely $\hat\theta$ is strongly correlated, consider using a confidence region rather than separate confidence intervals for its components. $\endgroup$ – whuber Sep 25 at 12:39
1
+50
$\begingroup$

Statistical independence/dependence does not make reference or concern itself with whether or not the data comes from the same experiment but about the data generating process. Independence in statistics is whether or not the data points influence each other, not whether or not they come from the same experiment. So your concern of "What troubles me here is that, these additional data points are not "independent" in the sense that, coming from the same experiment, we can expect them to be dependent on the previous data point." is not a concern in general but likely because of your specific experimental setup.

That is it's a different concern to say "my data are all from one experiment and so I might have a problem" than to say "my data are not independent". In your case you have both problems. However, the only way to address the limitation of "my data is just from one experiment" is to run more experiments. However then you get into the issue of "all my data is just from one experimental apparatus" so you need to find another experimental apparatus... so on and so forth. Another way to address this is to say that the all from one experiment concern addresses "systematic" error and the "data are not independent" concern reflects the concern that your uncertainties are miscalculated from that experiment (i.e., you are misestimating the "random" error.

Furthermore, for these models and this setup that you are doing I would argue that "degrees of freedom" is an ill defined concept and so you just shouldn't worry about it. The trickiness with defining it for your particular experiment is that it isn't clear in how many ways you can vary/change your system independently. I guess my answer is: you're asking a good question and the answer is unfortunately that the concept just doesn't really apply.

You are doing diffusion experiments and I have some familiarity with them though likely in a very different context and it sounds to me like you are measuring how quickly oxygen diffuses into a material (if this is wrong then your model will need to be more complicated). Let's go with measuring oxygen uptake in a material to measure the diffusion coefficient. So there are some possible parameters that your model is interested in fitting: the diffusion coefficient and the measurement error.

Instead of fitting a curve to your data as you have done now which leads you to all sorts of nasty considerations such as what happens when the data are not statistically independent (that is the data depends on other data points; in your case points depend on the value before them). I would propose you setup a physical model of your system and then fit the parameters using a bayesian model. The basic idea I am proposing is as follows:

1) write a physical model that generates the data from a diffusion coefficient 2) use a MCMC sampler to fit the diffusion coefficient and measurement error that best fits your data along with the uncertainties

For the MCMC sampler you will need to use a likelihood function and for that I propose weighted square deviations:

$$L = \sum_i {(y_i-s_i) \over \sigma^2}$$

where $$s_i$$ are the simulated data that correspond to each $$y_i$$ and $$\sigma^2$$ is the measurement error that you are fitting from your data.

Assuming that the MCMC sampler does it's job and finishes you should get both the parameter values and uncertainties without having to explicitly worry about degrees of freedom.

Here is some example code for a similar situation: https://github.com/bblais/pyndamics

$\endgroup$
  • $\begingroup$ Thanks. I would say that in a single dynamic experiment data points do influence each other as the value at t+1 is conditioned by the value at t $\endgroup$ – Toulousain Oct 11 at 7:06
  • $\begingroup$ Yes, it is gas diffusion, not really oxygen but it is a similar setup anyway $\endgroup$ – Toulousain Oct 11 at 7:07
  • $\begingroup$ So the thing about the dependence is that the data points are not dependent once you know the time at which the data point is taken because all you have at that point is the measurement error. What I mean is this: the data point at t+1 is not dependent on the data point at t because you have measurement error in your setup. The true concentration at t+1 is dependent on the true concentration at t, but once you know what which time you have sampled you can calculate that from the actual physical model of the system and so that dependence in removed when thinking about the measurement error. $\endgroup$ – Patrick Oct 14 at 13:32
  • $\begingroup$ I forgot to answer this in my post but it is another reason why I like the approach of specifying a physical model and then adding measurement error on top of it. A simple example of this is consider the case of a linear regression: The y value is dependent on the x value but once you know the x value and predict the likely y value the distance from the line is independent of the x value. $\endgroup$ – Patrick Oct 14 at 13:32
2
$\begingroup$

What you're dealing with here is a common task in the field of system identification: You're trying to estimate one parameter $D$ (diffusivity) of your model $C(D, t)$ (which is hypothesized to be the true model, normally found by applying first principles) based on measured data $C_m(D, t)$:

$C_m(D, t) = C(D, t) + n(t)$

Where $n(t)$ is measurement noise. $n(t)$ should be independent of $C(D, t)$. Furthermore, very often it is assumed that $n(t)$ is white noise with a constant variance $\sigma$. Under these assumptions, all $N$ samples of $n(t)$ are iid and you can construct confidence intervals for your estimation of $D$ with $N-1$ degrees of freedom.

So the main point is that your intuition is correct: The $N$ samples of $C_m(D, t)$ are not idd due to the function $C(D, t)$ being deterministic, but the $N$ samples of measurement noise $n(t)$ should be iid.

More details can be found in books about System identification. The classic reference is: Lennart Ljung: System Identification — Theory For the User, 2nd ed, PTR Prentice Hall, Upper Saddle River, N.J., 1999.

$\endgroup$
1
$\begingroup$

Suppose that your instrument measures $$y=f(D)+n$$ where $f(D)$ is the exponential diffusion curve that you plotted and $n$ is the measurement noise. Since you are talking about a single experiment we assume that $f(D)$ is deterministic and only the measurement noise is random. When you are talking about independence and degree-of-freedom, it concerns $n$, not $D$.

You wrote "the evolution of a deterministic system depends on the previous state. Therefore, the concentration at t+1 will depend on the concentration at t. Eventually, all the data points depend on the initial one."

This does not make any problem. This actually enables us to use all the points in the experiment to estimate one single parameter $D$, assuming a model function $f$.

In the case of a single experiment assuming that all the uncontrolled conditions are at a single value, the confidence interval only concerns the measurement process. Every measurement equipment has a bandwidth: it can produce a limited number of independent observations in a time span. Hopefully, the bandwidth of your measurement equipment is large enough such that it can measure $f(D)$ without any distortion. But we like to make as many observations as possible to improve the accuracy of the estimation.

You need to answer this question: "Do you already see bandwidth limitation in $n$?

You can calculate the residual of your estimate and study their autocorrelation function or their power spectral density. In the case that you see a strong correlation, you can calculate an equivalent number of independent samples and adjust your estimate of the variance of parameters. For example, is your collect 100 samples per second for a duration of 1 second then you have 100 samples. If you analyze the residuals and see that the bandwidth of the data is 25Hz instead of 50 Hz it means that half the samples are effectively redundant and then you need to multiply your variance by 2. Or you can downsample your signal to remove the correlation. If you don't see any correlation in the samples then the degree of freedom would be the number of samples minus one. But the confidence interval concerns the measurement process not the diffusion constant.

The fact that confidence interval is reported from a single experiment, however, is troublesome in my view, unless the propose of the experiment is to demonstrate the accuracy of the measurement apparatus, not the diffusion constant. We report the number because it supposed to be reproducible. So at least given all your conditions and instruments if someone repeats the experiment then she should be able to get similar numbers and we report confidence interval to make possible to measure similarity.

Also a side note About selecting the point: When estimating the parameter in a decaying process, e.g.: $$y = A - B \exp(-Dt) + n$$ The points close to $t=0$ are the most important ones. Since as $t$ increase the signal component to noise ratio becomes smaller.

$\endgroup$
  • $\begingroup$ Thanks. That's an approach I haven't thought of. My only concern is, would autocorrelation only mean "sampling redundancy"? Let's imagine that the model has a larger mismatch at large t, then the autocorrelation would be represent model mismatch, right? That would make it more useful even! But I want to make sure I understood your answer $\endgroup$ – Toulousain Oct 13 at 16:49
  • $\begingroup$ If the model has a larger mismatch n for large t then that should be modelled in as different variance for samples of n. The auto correlation of n shows the dependency of samples of n. $\endgroup$ – Hooman Oct 13 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.