3
$\begingroup$

I have two questions below. I have seen people doing sensitivity analysis in observational study papers for the model to check sensitivity to assumptions in bayesian context for selection of priors. I am not sure what is exactly sensitivity analysis as it has not popped up in the books I have read other than those involving bayesian methods. Strangely, I was never formally taught anything on sensitivity analysis.

  1. When should one perform sensitivity analysis? Is there any good theoretic and applied book on sensitivity analysis?

  2. How should one do sensitivity analysis in biostat in general?

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ Question (2) is perhaps too broad and too tangential for this site. (Consult a textbook, such as Gelman et al.) But (1) looks on topic. $\endgroup$
    – whuber ♦
    Jan 11 at 20:37
  • $\begingroup$ @whuber Which Gelman's book were you referring to 2 here? Gelman's BDA book seems to be related to Bayesian context in general. $\endgroup$
    – user45765
    Jan 11 at 20:49
  • $\begingroup$ Yes, that's the one. It is relevant. Why should biostatistics require exceptionally different applications of statistical methods? $\endgroup$
    – whuber ♦
    Jan 11 at 20:54
  • $\begingroup$ @whuber I see. Thanks. $\endgroup$
    – user45765
    Jan 11 at 20:57
  • 1
    $\begingroup$ Yes, these are all sensitivity to different assumptions, so they would not all be covered in a single textbook. There may be other resources on how to do that, but each sensitivity analysis will be specific to the assumption being tested. There is likely no singular resources for this, as the analyses will be spread across the literature where each assumption is most often invoked. If you ask specific questions about sensitivity analyses for specific assumptions, we may be able to point you in the right direction. $\endgroup$
    – Noah
    Jan 11 at 21:23

1 Answer 1

Reset to default
2
$\begingroup$

In my experience sensitivity analysis in statistical settings generally means some variation on: I estimated the model again, by altering one or more researcher-selected parameters, in order to understand how the assumptions enacted by these choices affect my results.

Example from analysis with missing data
Suppose one has a dichotomous (0/1) variable in one's model, and some of its values are missing. Perhaps you used analysis of only complete cases, or some imputation strategy to address this. You might also add a sensitivity analysis whereby you:

  1. replace all missing values with a 0, and re-estimate your model, and
  2. replace all missing values with a 1, and re-estimate your model.

The results of this sensitivity analysis might in a simple case bracket the largest effect data missingness can have on the results of your analysis strategy.

Example from analysis with uncertainties and biases
In research on rates of homicide of transgender people in the US, I was presented with overlapping biases towards under-count in the numerator, and uncertainty about the denominator for homicide rates, and proceeded to estimate homicide rates by:

  1. assuming no under-count, and assuming under-count at rates 0.2, 0.5, and 0.8 to create my numerators, and
  2. using (and interpreting) three different prevalence estimates from the wider literature to create my denominators.

With twelve estimates per homicide rate, I was able to note which qualitative patterns depended strongly on these assumptions, and which depended weakly or not at all on the assumptions.

Speaking to sensitivity analyses on Bayesian priors
There are distributions of prior belief, including distributions in expert opinion, and previous published results. One may repeat one's analysis using values at the extremes of these distributions, and at the center, or aligned with the centers of modes of these distributions to explore the consequences for model results given different researcher assumptions.

What do with the results of such a sensitivity analysis?

  • Combine the results as a model ensemble; describe the distribution of results
  • Examine which results are qualitatively or quantitatively robust to varying assumptions, and which are strongly dependent on varying assumptions. The latter may indicate that the nature of that which you study varies depending on which assumption holds true, or it may indicate poor prior knowledge, which is important to nail down firmly to get valid model results.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.