Let's say that I have a linear system (e.g. an electrical circuit) and I am trying to estimate the values of parameters in the system (e.g. the resistance, inductances, and capacitances). I do this by measuring the response of the system to some input signal (e.g. a step function). If the noise in the measurement were uncorrelated and Gaussian, then one obvious approach to parameter estimation would be to use the maximum-likelihood estimator, which would reduce to a (non-linear) least square fit. Simple enough.

However, if the measurement is very sensitive, so that the dominant source of noise comes from compoments internal to the system (e.g. Johnson noise in a resistor), the noise at the output will be correlated. This means that maximum likelihood estimator for the parameters will not reduce to a simple least squares procedure.

I'm sure that there is a set of statistical practices and literature devoted to parameter estimation for systems like this, but I'm not sure where to start looking. Can anyone point me in the right direction?

Update I just want to clarify that although the system is linear in the sense that a linear superposition of two signals is also a valid signal, the output of the system as a function of time is certainly not linear. In general it would be some sum of exponentials, where the coefficients in front of each exponential are function of the various time constants.

  • 1
    $\begingroup$ If the errors were correlated over time you could model the noise component as a mving average with the coefficients of the moving average determining the autocorrelation function for the noise. Model parameters for time series which include the coefficients of the moving average terms can then be fit using maximum likelihood. If the data do not form a time series there still may be a way to model the noise term so that correlation is part of the structure. $\endgroup$ – Michael R. Chernick Sep 18 '12 at 22:11
  • $\begingroup$ Michael - I don't understand exactly what you are saying, but a feature of this system is that the autocorrelation function of the noise is a known function of the parameters to be estimated. I figure that this must help - the noise itself contains information about the parameters! $\endgroup$ – Dan Becker Sep 18 '12 at 22:21
  • $\begingroup$ One thought I just had is that since the system is linear, the problem is likely simpler in frequency space, and in fact it may be the case that the noise is uncorrelated in frequency space. $\endgroup$ – Dan Becker Sep 18 '12 at 22:25

So if you have a time series model for the measurements

X(t) = f(previous Xs and ts) + N(t) where N(t) = noise at time t.

then you can write N(t) =e$_t$ + a$_1$ e$_t$$_-$$_1$ +a$_2$ e$_t$$_-$$_2$ +...+ a$_k$ e$_t$$_-$$_k$

where the e$_t$s are iid random noise (i.e. white noise) Then N(t) is correlated noise with autocorrelation determined by the a$_i$s (k could be just 1 or it could be 2 or higher).

Given a model like this you can write down the likelihood and find the maximum. Being that you say f is linear this is solved via conditional least squares.

  • $\begingroup$ OK, that makes sense! In my case the a_k are known functions of my parameters, but the response function is non-linear (see update above) so I guess conditional least-squares does not apply. But I can imagine how I would write some code to evaluate MLE using the above model. $\endgroup$ – Dan Becker Sep 18 '12 at 23:14
  • $\begingroup$ Having said that, I still think that there must be literature out there specific to this this problem; people have wanted to measure parameters in electrical circuits for a long time! $\endgroup$ – Dan Becker Sep 18 '12 at 23:15
  • $\begingroup$ A standard approach seems to me to use a Kalman filter, extending the state vector to account for the correlation in the noise. You may want to look, for instance, Gibbs, B. Advanced Kalman Filtering, Least-Squares and Modelling, Wiley, 2011. You have non-linearities, though, which may complicate things a bit. $\endgroup$ – F. Tusell Dec 18 '12 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.