2
$\begingroup$

This question is coming from a business background.

I want to focus here on issues that occur within a business (or perhaps academic) background which are usually difficult to quantify but it is hypothesised that they could lead to bias in the statistics produced by an analyst (as part of a report, for example). A simplification of these scenarios is typically when an analyst will be given/request a number of data tables alongside a question that needs to be answered/explored. The analyst will then uncover issues within the data when carrying out analysis.

I will give some examples of such issues:

  1. An organisation measures something using one classification system but starts to operate a new classification system. The old classifiers have to be mapped to the new classifiers but it is a messy many:many mapping. There is now some uncertainty when comparing the time period with the old classification system with the time period with the new classification system (i.e. is a sudden peak a result of the classification system or an interesting phenomenon)

  2. An organisation stores the history of different data tables at different intervals of time. This means when bringing the two data tables together, some data is stored at annual snapshots, other data is stored in weekly snapshots. Some assumptions have to be made as to how to match the two (i.e. if past July, use the annual snapshot for the next year).

  3. It is known there is under-reporting / data is missing not at random but there is no scope to understand why (least not for the impending piece of analysis).

  4. Relational data mapping. For example, there are 3 names attached to one internal investigation and one decision attached to the internal investigation. So we have ultimately don't know what relationship exists between the names and the decision (did the decision relate to all 3, 2, 1 or 0 of them?).

  5. A key field is a subjective assessment by different people in the organisation but I don't know who made which assessments and how biased (or otherwise) some people may be.

In my eyes, these issues increase the uncertainty around any statistics produced. But how can one incorporate this uncertainty so that, on a particular report, the issues don't sit in one section of the paper and the results in another section (such that a busy member of management will oft skip the issues and head straight to the results)?

As an extension of this, how does one decide when a piece of analysis has so much uncertainty that it detrimental to provide it as part of making a decision?

I appreciate this is quite a high-level question and would definitely appreciate feedback on how I can give it more scope.

$\endgroup$
1
$\begingroup$

If you really want a colleague to avoid putting too much trust in a point estimate, then you can report a confidence interval or credible interval first, and save the point estimate for later (or skip it altogether). Similarly, if you want to express the amount of uncertainty that surrounds a certain method of measuring some characteristic or of collecting or processing data, you can present alternate methods side by side. Those who might be inclined just to size up the key fact on a given topic will be struck by the lack of any one key fact to seize upon. This will likely result in a) more nuanced consideration of the complexities you seek to present, b) impatient calls for a simpler presentation (which you can try to turn around towards [a])*, or c) delegation of the issue, leaving it up to those who are willing to struggle with the complexities.

As to the "how does one decide" question, this is probably too broad for any single answer, and doesn't likely have any answer that the community would judge as correct.

*Everything should be kept as simple as possible, but no simpler. Albert Einstein

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I'm not sure, if incorporating uncertainty due to lack of solid business architecture, non-optimal business processes and data quality issues is the best approach to the described situation. While you can certainly use some statistical approaches to improve the situation a little, I think that the right approach to such problems is a top-down one. I would consider the following measures:

  • create or improve business architecture (BA) for the organization;

  • create or improve IT and data architectures, both of which should be aligned with BA and business goals, in general, as well as between each other;

  • optimize business processes, based on the above-mentioned changes (for example, via business rules);

  • for data architecture, pay attention to master data management, data quality, standardizing statistical analyses, reproducible research and establishing data analysis workflows (you can read more on that in my answers on Data Science SE site, linked in this gateway answer).

This is just a high-level overview of the approach that I would recommend. Anything beyond that requires knowing details about the organization and most likely calls for some consulting effort.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.