This question is coming from a business background.
I want to focus here on issues that occur within a business (or perhaps academic) background which are usually difficult to quantify but it is hypothesised that they could lead to bias in the statistics produced by an analyst (as part of a report, for example). A simplification of these scenarios is typically when an analyst will be given/request a number of data tables alongside a question that needs to be answered/explored. The analyst will then uncover issues within the data when carrying out analysis.
I will give some examples of such issues:
An organisation measures something using one classification system but starts to operate a new classification system. The old classifiers have to be mapped to the new classifiers but it is a messy many:many mapping. There is now some uncertainty when comparing the time period with the old classification system with the time period with the new classification system (i.e. is a sudden peak a result of the classification system or an interesting phenomenon)
An organisation stores the history of different data tables at different intervals of time. This means when bringing the two data tables together, some data is stored at annual snapshots, other data is stored in weekly snapshots. Some assumptions have to be made as to how to match the two (i.e. if past July, use the annual snapshot for the next year).
It is known there is under-reporting / data is missing not at random but there is no scope to understand why (least not for the impending piece of analysis).
Relational data mapping. For example, there are 3 names attached to one internal investigation and one decision attached to the internal investigation. So we have ultimately don't know what relationship exists between the names and the decision (did the decision relate to all 3, 2, 1 or 0 of them?).
A key field is a subjective assessment by different people in the organisation but I don't know who made which assessments and how biased (or otherwise) some people may be.
In my eyes, these issues increase the uncertainty around any statistics produced. But how can one incorporate this uncertainty so that, on a particular report, the issues don't sit in one section of the paper and the results in another section (such that a busy member of management will oft skip the issues and head straight to the results)?
As an extension of this, how does one decide when a piece of analysis has so much uncertainty that it detrimental to provide it as part of making a decision?
I appreciate this is quite a high-level question and would definitely appreciate feedback on how I can give it more scope.