# Identify the confidence of the impact of a proposed improvement initiative

Assume I have a manufacturing process that involves a moving train. It has failures of certain types like brakes and steering and also the weather. However, we can not do anything with regard to weather. Furthermore, there are more Brake failures than steering failures, we decided to tackle the brake failures.

Using an engineering intuition we can assume that we can get rid of 50% of those failures by making some changes to the engineering design. However, as said previously, this is only an engineering intuition; hence my question is there a statistical technique to access the confidence that our improvement initiative will indeed reduce the brake failures.

I understand that once we have implemented the initiative we can use t-tests and ANOVAs to ascertain the same. But my question is that is there anything I can leverage prior to embarking on the engineering change.

Data for example could look like below:

Equipment   start_time     end_time        Reason            duration
Train1      201201130415   201201130445    Brake failed      0.5
Train1      201201130221   201201140336    Steering failed   1.25
Train1      201201130119   201201180349    Brake failed      2.5
Train1      201201131018   201201211148    Brake failed      1.5
Train1      201201130815   201201250945    Weather bad       1.5


I think for a more specific answer we probably need you to give us more details about the data, the intervention, other possible data sets, etc. However, based on the information in the question I will try to give you an idea of what an answer might look like.

TLDR; it depends on how strong assumptions you are willing to make. i.e. are you willing to write a model characterizing the expected relationship!

There are a couple of approaches that you should be aware of:

This question is closely related to the literature on causal inference. However, the classical atheoretic approach to the causal identification problem requires both treatment and control data where treatment data gives us information on the outcome post-intervention.

So from the point of view of that literature, the answer would be no. Indeed, when you say you want to go from intuition to statistical confidence the only way to truly do this is to have data that gives us some information about the planned intervention. Right now you have essentially asked us to "fly blind" without any information about why some intervention should work or what evidence there is that it does work.

However, all hope is not lost.

Scenarios like this do sometimes occur. One example is in economics. An economist might be asked to find the effect of tax reform on some measure of a countries welfare. Now, if this reform had been done in the past the researcher may attempt to use some of these causal inference techniques to try to evaluate the policy. This is part of the policy evaluation literature. Researchers like this approach because it usually means they can avoid making very strong assumptions. Hence, why I earlier called it "atheoretic". But let us be clear, they do need to make assumptions. The idea in this method (and in the next method I will discuss) is that to evaluate a policy intervention we want to identify a counterfactual. In your case, this would be what happened to the same train, in the same situation, if it both had or did not have the engineering intervention. It is called a counterfactual because it is fundamentally impossible to know both at the same time. However, if we did know both then we could look at their difference to determine the effect of the counterfactual. Thus, in order to identify the counterfactual (even when we have access to data), we need to make some assumptions.

Now, in your case, we do not even have data with the intervention. Thus, our assumptions better be strong! Going back to the economics example if a researcher did not have access to a previous reform or data what they may do instead is to write what is called a structural model. The idea is that the researcher uses economic theory (or in your case engineering knowledge) about the problem to write a model of how the intervention will affect the outcome. This can be done with formal mathematical expressions.

For example, in your problem, I did a quick google search and found this example of a paper that studies the braking systems assuming that failures are distributed according to a Poisson process. The idea is that this stochastic process will govern when a failure occurs. Now the process has a parameter $$\lambda$$ that controls the rate at which failure occurs. Perhaps you assume,

$$\lambda= constant \times \text{brakeTolerance}$$

where your intervention is now changing the tolerance of brakes in some way. Now you could simulate data from this distribution under different brake tolerance levels and determine how tolerance will impact the rate at which there is a brake failure.

Obviously, this is a very simplistic model! The idea is valid though, you apply your engineering intuition and knowledge to develop a model that relates the intervention you seek to implement to the outcome of interest and study the comparative statics (as we call them in economics) of the problem. Basically, you have made a strong assumption (on the parametric form of the intervention model) to determine the effect of your intervention.

Now, I will leave you with a bit of a philosophical quandary. Can we consider this method statistical?

The reason I pose this question is that instead of finding data and conducting inference given the data we have instead decided on a data generating process (DGP) and studied this DGP. In the example above the DGP involves "random processes" and so perhaps from that perspective, it is statistical. One could also see this as a Bayesian approach in some sense, where our prior is the model that we imposed. In economics, we call this the reduced form vs. structural model debate. I personally tend to believe that almost all reduced-from is structural because even when we run a simple linear regression we must make assumptions (including often a linearity assumption) in order to do valid inference and even get identification. I think the point is that we no matter our methodology we need to be careful about and identify the assumptions that we need to make.

Here is a nice set of notes on the structural vs. reduced form debate in economics in case you are interested!