# Test if one of many factors improved performance

We give customer surveys and obtain millions of customer satisfaction scores every year. We have 100 measurable factors related to the scores, such as service type, date, zip code, etc. One department worked on improving one of the factors (e.g., checkout time) this year and would like to see if their work positively affected the scores. But we would like to exclude the impact of other factors. This is the method I can think of:

1. Create a predictive model (e.g., random forest) with the 99 factors besides the one got improved as the predictor variables, and customer satisfaction scores as the response variables.

2. Train the model with last year's data (before improvement), and use it to predict this year's scores.

3. Somehow check if the model is accurate enough (maybe hold some of last year's data to test), and check if the actual scores this year are significantly higher than the predicted ones.

Is there any standard method to do this kind of study?

• What outcome have you measured and what are your replicates (e.g. one measure taken repeatedly over time or several measurements taken from clusters or individuals). See stats.stackexchange.com/tags/analysis/info for advice on how to edit this question. Dec 14, 2015 at 21:39

Your hypothesis concerns whether a particular intervention improved some performance. As stated, we need not create a prediction model necessarily. You are simply trying to estimate a conditional mean difference in the outcome under the two settings: one where the intervention is applied, and the other where it isn't. This may indeed be as simple as a plain old t-test. Controlling for other, independent determinants of the outcome, however, will increase precision (and, therefore, the power to detect if the intervention was effective).

If repeated measurements were drawn over time, then you are fitting a basic pre-post type of model using historical trends to infer the typical "intervention free" outcomes. If you implemented the intervention in an A/B setting, then there are groups of individuals representing either scenario.

In any case, there may be things influencing the observed outcome having little to do with the intervention, such as autocorrelation or secular trends. Controlling for these in a regression model allows you to make true "apples-to-apples" comparisons.

So, depending on the nature of the "influencing" factors, you can always use adjustment in a multivariate regression model.

Alternately, if there are 100s of such predictors, controlling for all of them simultaneously is impossible if the sample size is modest (perhaps, under 1000) due to multicollinearity, loss of efficiency, and other considerations. In this case, a sensible approach is using propensity score adjustment. This is a hybrid of prediction and inference. You use these 100s of predictors in a binary regression model to predict who is likely to have received the intervention and who is likely to not have, create a propensity score, and adjust for it in an analysis including the binary intervention indicator. Typically, these are still very highly collinear, so loss of power is still a consideration. The advantage of propensity score adjustment is that you may develope such scores using the machine learning techniques of feature selection to create a parsimonious model when predictors outnumber observations.

• Many of other factors are changing over time. For example, many customers may shift from category A to category B for a certain variable, and therefore tend to give a higher score. So it is probably necessary to control them. In this case, is propensity score method the right way to go?
– Fan
Dec 14, 2015 at 22:33
• @user2316040 so it confirms that there is a time series component to these data, and also that you have panel data, and also that the outcome is continuous. Very important to know. That other factors are changing over time is unimportant. The salient thing is whether they influence the outcome. If they do, you must adjust for them. Dec 14, 2015 at 22:35