3
$\begingroup$

I have very basic knowledge of stats so my question may sound very simplistic. I have a large time series of measured data and have calibrated a model with five parameters to make predictions. I would like to test the sensitivity of my model but I am not sure what statistical method to use. I've been reading about sensitivity analysis, however the output of my model is a time series with fluctuations and a simple sensitivity analysis does not seem appropriate since it looks at individual output points and not the entire series. Any recommendations on methods or approaches?

$\endgroup$
2
$\begingroup$

One solution is to reduce the dimension of your output by using a carefully chosen projection basis (see [1]).

If no particular type of basis comes to mind when looking at the data, you could apply principal component analysis and use the scores of the first few components as new output variables (see [2] [3]).

References

  • [1] McKay, M. D. (1988). Sensitivity and uncertainty analysis using a statistical sample of input values. Uncertainty analysis, 145-186.
  • [2] Lamboni, M., Makowski, D., Lehuger, S., Gabrielle, B., & Monod, H. (2009). Multivariate global sensitivity analysis for dynamic crop models. Field crops research, 113(3), 312-320.
  • [3] Lamboni, M., Monod, H., & Makowski, D. (2011). Multivariate sensitivity analysis to measure global contribution of input factors in dynamic models. Reliability Engineering & System Safety, 96(4), 450-459.
$\endgroup$
1
$\begingroup$

Professor Edward Leamer in his 1985 article Sensitivity Analyses Would Help, says the following:

What we need instead are organized sensitivity analyses. We must insist that all empirical studies offer convincing evidence of inferential sturdiness. We need to be shown that minor changes in the list of variables do not alter fundamentally the conclusions, nor a slight reweighting of observations, nor correction for dependence among observations, etcetera, etcetera.

Following along similar lines to Professor Leamer, I would make two suggestsions about how to perform a sensitivity analysis.

The first is to investigate whether or not the results of your model are sensitive to changes in the data set.

The second is to investigate if your model's results are sensitive to small changes in model specification.

Let me now expand on both of these approaches.

The first approach is to fit subsets of the data. That is, estimate the model over the entire data set and then re-estimate the model using subsets of the data. For example, you could divide the data set into a 50-50 split and estimate the same model using each half of the data. The next step would be to carry out statistical tests (e.g. t-test) to see if the coefficients from the two data sets are significantly different.

A second way to do it would be to drop, say, 10% of observations, re-estimate the model and see if the coefficients are within $\pm$ 0.1 of the coefficients generated by the model estimated over the entire data set.

The second approach involves making small changes to the specification of the model, estimating the re-specified model, and investigating whether or not the coefficients change drastically. This approach corresponds to Leamer's changes in the list of variables. Now, I understand that "drastically" is not a very scientific way to put it, but it would do no harm to keep an eye out for sign changes and big swings in the values of the estimated parameters.

An example from macroeconomics is that not only is the sign of a fiscal multiplier (link goes to wikipedia) important, but it's also important to try to discover whether or not the multiplier is greater than one or less than one. Depending on the data you're working with, you may want to watch out for changes like that; a small change could have a significant practical meaning, i.e. a multiplier of 0.99 implies something very different about the effect of government spending than a multiplier of 1.01.

Hopefully even a fraction of this is helpful to you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.