Generally speaking, there is ample literature on characteristics of diagnostic tests (eg the
Joanna Briggs Institute Guidelines or the Cochrane Methods Section on Diagnostic Test Accuracy Studies) detailing which statistics should be computed in a diagnostic test accuracy study.
Briefly, you should first identify true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN):
Then, you can compute:
- prevalence, defined as (TP+FN)/(TP+FN+TN+FP);
- sensitivity,defined as TP/(TP+FN);
- specificity, defined as TN/(TN+FP);
- diagnostic odds ratio (DOR);
- positive predictive value (+PV);
- negative predictive value (-PV);
- positive likelihood ratio (+LR), defined as sensitivity/(1-specificity);
- negative likelihood ratio (-LR), defined as (1-sensitivity)/specificity;
- area under the curve (AUC) of the receiver operating curve (ROC).
The most accurate and robust statistics are +LR, -LR, and AUC, as they are less dependent on disease prevalence. In addition, diagnostic tests may have prognostic features, and you could also envision randomization to a diagnostic test.
Let's now focus on your questions:
Is it an error to change the threshold?
It could be risky, amounting to data massaging, unless you have sound clinical reasons to do that. Remember that the threshold you are proposing to readers is one only and should be stable for subsequent clinical application.
How can I describe my data in terms of its ability in diagnosis?
That's difficult to say without looking at the actual curves. It might be you have few cases and a few misdiagnoses are very impactful.
What could cause this?
As above, if your sample is small with few or many diseased cases even few misdiagnoses can be impactful on sensitivity and specificity.
Would it be reasonable to say that "the test seems to have a reproducible specificity but many cases may be missed" or does this mean that the validation completely failed?
I would also compute DOR, +LR, and -LR. If they are consistent, you can simply state that in your sample sensitivity is not the appropriately stable and precise statistics to describe the diagnostic accuracy of your test.