Good practice for statistical analysis in a business environment

Question

(While I realise this isn't strictly about statistics, it is about the dissemination of statistics in a business environment so I have assumed it is still within the topic range of CV)

A brief bit of background:

Our business environment (and I suspect other environments) have a support function who specialise in statistical analysis and research. We work closely with Business Intelligence and are commissioned by other departments to produce pieces of work. In effect, the data, analysis and conclusions don't belong to us: we collect data, perform analysis and draw conclusions for the commissioner to use within their work.

What I want to do:

Currently, we run quite a laissez-faire approach. An individual from the support function is assigned when work is commissioned, data is collected (or extracted, if it exists, by Business Intelligence), analysed and the final set of conclusions are sent to the commissioner. This has been loosely justified on the basis that it is not the commissioner's role to read through the analysis; it is our role as a support function to ensure we provide the right analysis for the questions/topics the commissioner wants to explore.

I want to invoke a little more structure on the approach to make

a) our analysis of a higher quality;

b) provide defensibility when our analysis may lead to bad decisions; and make

c) our analysis more transparent so we aren't viewed as a 'black box' that takes data and spits out results.

My initial thoughts have been to:

1. Produce a technical document with every piece of work that justifies the approach taken, the assumptions made, the issues found, the uncertainties that exist etc. While this won't necessarily be read by everyone, it should be used as a means to explain to the commissioner the consequences of using the conclusions drawn. This transfers some of the risk to where it feels like it should belong: with the commissioner.

2. Restrict all analysis to a package such as Stata, SPSS or R and require a full set of code to be produced alongside the technical document. All of us have a habit of using Microsoft Excel for some types of analysis (bad habit more than anything). However, Excel doesn't promote easy reproducibility of analysis. This helps defend the support function when our analysis is questioned, creates transparency in our approach but also makes the role of (3) much easier:

3. Assign a reviewer to every piece of work who must 'countersign' the work before it is sent to the commissioner. By countersigning, it distributes the integrity of the analysis across 2 people and encourages them to work together (2 heads are better than 1). This should improve the quality of analysis and also provide some defensibility.

Are there any other facets of good practice that can be applied in a business environment of this kind?

Answer 1

My advice in two words (TL;DR mode): reproducible research.

For more details - largely not to repeat myself - let me refer you to my relevant answers elsewhere on StackExchange. These answers represent my thoughts (and some experience) on the topics:

• data cleaning: https://datascience.stackexchange.com/a/722/2452
• reproducible research: https://datascience.stackexchange.com/a/759/2452
• reports vs. dashboards: https://datascience.stackexchange.com/a/907/2452
• data analysis workflows and EDA: https://datascience.stackexchange.com/a/1006/2452
• big data and R: https://datascience.stackexchange.com/a/780/2452

Final note (sorry, if you find it obvious): regardless of the type of your business environment (which is unclear, by the way), I would recommend to start from business side of things and create a data analysis architecture, which (as all IT-related) should be aligned with business architecture, including business processes, organizational units, culture and people. I hope that this is helpful.

UPDATE: In regards to creating a new or improving an existing data analysis architecture (also referred to as data architecture, in enterprise architecture terminology), I thought that these two sets of presentation slides might be useful as well: this and this.

Answer 2

In banking the modelling must comply with model risk management guidelines, such as OCC 2011-12. I think it's an interesting document even if you're not in banking.

MathWorks has this article on modeling standards.

Since modeling involves writing software in one form or another I use elements of software development methodology, particularly when it comes to testing and unit testing. I also employ software configuration management tools such as SVN. There's a lot that modeling teams can learn from programmers in terms of managing complex software projects, such as issue tracking systems and CMS.

One of the most important things is the methodology and process, model development life cycle. Create the guideline of how to develop the models, and test them, list the standard tools and test etc. For instance, pick one or two goodness-of-fit tests, and use them everywhere.

Create templates of everything: modeling scripts, white papers, presentations etc. For instance, I have the templates in LaTeX for all documentation, so our white papers look very similar and everyone knows where to look for information. We have standard sections, such as descriptive statistics and standard columns in them such as kurtosis, first and last observation date etc.

Have the lab journal. This is one thing that hard science people should have learned in PhD: to keep a diary of all the research, ideas and especially decisions. When you decided to use ARIMA instead of GARCH, record it in the lab journal and describe why you made the decision. Down the road people tend to forget the rationale behind the decisions, so it's important to record them. Unfortunately, folks from social sciences backgrounds have no habit of keeping the lab journals, it's a problem.

Answer 3

Another aspect of good practice is discipline at the initial commissioning stage. This might include basic things like agreeing in writing what is required by the commissioner (to avoid misunderstandings and subsequent disputes) and clarifying who in the business has authority to commission work (a first step towards ensuring that the function is addressing real business needs and not just indulging anyone who has a bright idea).

Discipline in commissioning should also promote constructive dialogue prior to agreement on the work to be undertaken. Those commissioning may have a vague idea of what they need but have difficulty in formulating it precisely, or if they do offer a precise formulation it may not be what is most relevant to their business needs (for example, they might ask for an investigation of the reasons for a short-term fall in sales, when what they are really interested in are the longer-term factors driving sales). Statisticians and researchers may be good at formulating precise questions or plans of work, but less able to identify what will be useful to the business. There is I suggest a parallel with good practice in academic research which makes a distinction between research questions identifying fairly broad topics of interest and research hypotheses and aims within such topics which are specific enough to lead to well-defined research studies. Thus it may be helpful to think of the commissioners as generating the equivalent of the research questions and the statisticians and researchers as helping them to identify more specific work programmes relevant to those questions.

Answer 4

I think you have got part of your answer in the question - a "good structure" is key.

I am an engineer and have been working in roles that emphasise a similar application - where you are introduced to problems to provide assistance with analysing and improving the outcomes but are in an advisory rather than implementer role.

The best approaches, that I have seen, are ones that are not too prescriptive or loose to ensure the right amount of evidence that the work was done with diligence - which is what I think you are after.

Six Sigma (which is a bit of a dirty term in some places I have worked) and other methodoligies provide a framework for approaching, solving and embedding a solution. Because they are based on a framework, they can be audited. The key is to ensure that everyone is trained in the methodology AND have a good template that is auditable.

For example, you probably want the solutions to be of a standard - this is not defined by the program used but rather whether you can audit the steps of analysis used at a later date and be satisfied that the task was completed to a standard. Providing milestones - e.g. check points where you can audit will be easier than trying to audit at the end of the project.

Returning to Six Sigma, some approaches might be to audit at the Define stage, after Measure and Analyse, and finally at the conclustion (after Improve and Control).

Six Sigma is certainly not the best in all situations but I can recommend it as a potential starting point.