Design of experiment: Is there correlation between performance vs. employee attrition?

Background:

Assume company X has a cyber security incident team. One of their key performance/risk indicators is how fast they triage the security incidents. For the past 2 quarters, COO observed decreased amounts of incidents triaged within ideal/expected triage time.

Find out why there has been a decreased amounts of incidents triaged within ideal triage time.

Approach:

1. Employees come and go. There has been a continuous resignations. Test if decreased number of team members has correlation with triage time.

Problem:

I'm using simple linear regression. Triage time (in minutes) is response variable (y) and 'number of people joined/left' is explanatory variable.
Triage time data range is from 0~940000 minutes, and number of people is only from -4 to 2. -4 being when team lost 4 people and 2 being where they had 2 extra members from where they began the team.

Below is what it looks like (fake data), and I have 65000+ rows

The result is saying the model isn't significant.

Result:

Regression Statistics
Multiple R 0.004008821
R Square 1.60706E-05
Standard Error 642.2262936
Observations 65601
Significance F 0.30453744

I think maybe there is a better way to study relationship between employee attrition versus triage time.

Question is

Is it okay to compare the variables that one has much wider range (y) compare to another (x)? I'm thinking if I was supposed to perform any data transformation so I can somehow emphasize the employee attrition. Do you see anything wrong with the way I design the linear regression?

• Do you believe the model is wrong because you do not like the result or are there other reasons? Do you have some additional information about the data, that we are unaware of? Regarding the modeling, what model do you use to calculate the F statistic? Nov 13, 2017 at 9:05
• What is your n (number of incidents)? How were incident type and team_size modeled? Dummy-coded? If so, how many categories? Nov 13, 2017 at 12:26
• @MartijnWeterings The model itself is not wrong. But I'm being cautious of the possibility of me designing the poor model because this hypothesis is important. I used the simple linear regression. Some more detailed information about the data: I have 65601 number of data from 6/2016~now. It is consisted of 2 columns, indexed by incident observed date. 2 columns are 'triage time' and 'staffing power'. Triage time unit is minutes and it goes from 0-90000. For staffing power, it ranges from -4,-3,-2,-1,0,1,2. 0 means there's no change in staff#. 1 means one more staff, -1 means we lost a staff Nov 13, 2017 at 17:53
• Could you for yourself and for others reading the question make an overview of that data (and update this in the question). Plot distribution/histogram of triage time versus staffing power. It seems like the triage time is not a "nice" variable (ie with a Gaussian distribution). If your view of that histogram makes you suspect something then you can quantify it either using a more specific model (eg GLM or log-transformed OLS) or create bins and do some independence test like a $\Chi^2$-test. Nov 14, 2017 at 9:06

The short answer is: No, it is not a problem to use predictor variables in linear regression that are on a differently scale than the outcome. An extreme example is the use of dummy variables to represent factors: Dummy variables only contain 0's and 1's, but this is completely fine, as long as the other assumptions of your model hold — and here I can see problems with the scenario described above:

1.) The assumption that staffingPower and triageTime are linearly related may not hold. E.g. a staffing power of -2 to 2 may be fine, but at -3 to -4 the efficiency of the team may break down.

• Solution a): try what happens when encoding staffing_power as categorical variables (i.e. 5 dummy variables).
• Solution b): try what happens if you go non-linear (e.g. by polynomial expansion of your predictor staffing power)

(both of these solutions will also "emphasize" potential effects of employee attrition)

2.) The assumption of independent samples/no autocorrelation may not hold in your case, as the response time to incidents may be temporally correlated. This will not affect your regression coefficients, but may bias the error estimates. You can check for auto-correlation by using an auto-correlation plot, Durbin-Watson-Test, and/or by plotting the residuals of your regression against createDate (and then see whether any kind of pattern emerges other than random noise).

3.) The assumption of homoscedascity may not hold: Maybe your model predictions for long triageTimes are more variable than the predictions for short triageTimes? Check by plotting residuals against predicted values. If there is heteroscedascity you will see a "funnel" like relationship. Potential solution: transforms of triageTime (e.g. log(triageTime)).

• Thank you! I'm trying solution a, b! I'll let you know how they turn out Nov 15, 2017 at 17:09
• I'm trying out the solution A now. For this, do you think we still have to apply the outlier treatment? Data is skewed very much due to the extreme outliers Jan 11, 2018 at 17:52