Hypothesis testing business case study: does this make sense?

Question

My boss has a PhD in Math from a top 30 university; point being he's a bright guy but doesn't have a formal background in statistics. For background, I have an MS in Statistics but focused more on machine learning. He asked me to do a hypothesis test but it's unlike anything I've ever seen before so I'm wondering if it even makes sense. We are both data scientists.

Presently, our company has a system that automatically classifies phone numbers and by extension, their calls, as "bad" or "good." When the first call by a phone number is made through our system, we undertake an expensive manual investigation of various attributes related to the phone number to determine whether it is a "bad" or "good" phone number. The relevant aspect of this system is that there is a business rule in place where a previously classified "bad" phone call all subsequent calls labeled as "bad," if those calls are made within, let's say, 40 hours of the first call.

The stakeholders admit this 40 hour threshold is somewhat arbitrary and we've been tasked with finding a new threshold, supported by data. My boss said I should do this:

1. Build a dataset using call data from the past 4 months. Each observation is a phone call. A single phone number can make multiple phone calls. This dataset should only contain numbers that started off the 4 month period classified as "bad" and was later classified as "good" at any point afterward, in the same 4 month period.

2. Calculate the time difference between ANY two calls made within the same week. A week is defined as a time period starting on Monday 12am and ending 11:59pm on Sunday. Clearly, this restriction makes it such that the max "time gap" between any two calls is 168 hours. For example, phone number 1 making 3 phone calls will result in 3 rows of data that looks like:

    phone_number     time_gap
    phone number 1   hours between call 1 & 2
    phone number 1   hours between call 2 & 3
    phone number 1   hours between call 1 & 3

3a. Create a new dataset that is a filter of the above data set such that time_gap is in [20, 50].

3b. Create a new dataset that is a filter of the above data set such that time_gap is in [20, 70].

4. Find the mean of datasets 3a, 3b. Calculate the Z-score for this test statistic, using the formula:

   Z = (Xbar - mu) / (StD / sqrt(n))

where

Xbar = mean of a sample with replacement from 3a, 3b
mu = 40 hours, the current business rule threshold
StD = the standard deviation of a sample with replacement from 3a, 3b
n = size of the sample from 3a, 3b

---

My questions are:

1. Does this even make sense to do? In my mind, by restricting the data in steps 3a and 3b, we are kind of manufacturing the results for our sample mean and just trying to show that calls in this range are statistically significantly different from 40. Then, so what? This seems like cherry-picking our data and seems to definitely break the rule of: "don't throw away data you don't like just because you don't like it."

2. This whole thing doesn't make sense to me because we're just comparing two arbitrarily defined subpopulations of the "bad to good phone number" population. Doesn't it make more sense to find the mean number of hours between any two calls, between 2 different populations? For example, the "bad to good phone number population" and the "bad to bad phone number population"?

3. Does it make sense to take repeated samples from datasets 3a and 3b, i.e. bootstrapping to get confidence intervals for my mean? My boss saw that I did that and laughed, saying that it's pointless because doing so is basically just trying to prove the Central Limit Theorem.

4. Don't I need to check for normality of the underlying distribution of the time gap field?

Detailed responses are appreciated. I've already made the points above to my boss on a few occasions but he basically said I just don't understand basic statistics (which is possible, lol) and has become pretty exasperated at this point. Normally, I'd just let it go and do the work without much argument but we (read: I am) are going to have to present and defend these results to external statisticians and I'm honestly not a believer.

Answer 1

Although the boss's suggestion or the modeling approach suggested by @PaulG certainly might work, consider survival analysis too.

You are interested in the time to an event: the time of a switch from a "bad" to a "good" call type, starting from the first "bad" call from a phone number. Time-to-event analysis is the use case for survival analysis. Survival analysis would also properly take into account any "censored" phone numbers for which you never observe the switch to a "good" call type; omitting such "censored" phone numbers might lead to bias.

The reference time of 0 for each phone number would be the time of the first "bad" call. The event time (switch to "good") could just be the elapsed time to that switch, although you might want to model that as interval-censored (occurring sometime between the last "bad" call and first "good" call). Phone numbers that don't have an "event" would be noted as censored as of the last observed "bad" call (or whatever time limit for observations that you impose administratively).

You could use parametric models based on assumptions about the shape of the distribution of the switch times, go all the way to a completely non-parametric model as with clinical Kaplan-Meier curves, or incorporate other information in semi- or fully parametric regression models. The wealth of tools already available for time-to-event analysis would minimize the difficulty in developing a none-off analysis method and would certainly be easier to explain to outside statisticians.

Answer 2

What you need is essentially a good model for the "switching time" of a call from good to bad. When designing a model it is most useful to start from an ideal situation and then approximate that using the data at hand. The following are just my own thoughts, read them critically. :)

Ideal model

The scope is to statistically see if $\mu=40$ is a good value for the average time within which calls don't change their status from bad to good. Let $Z_{it}$ be the $t$'th call from number $i$ and $X_{it}$ the time between this and the previous call, i.e. $X_{it}=time(Z_{t})-time(Z_{t-1})$ ($X_{i0}$ doesn't exist), with $Z_{it}\in \{good,bad\}$. Ideally, we would know the true classification of each call, i.e. $Z_{it}$. Then, since we want to test only how fast bad calls turn to good calls, we would take all the times between consecutive calls which switched from good to bad, i.e. the set $\mathcal{X}_{bad \to good}=\mathcal{X}= \{ X_{it}:Z_{it-1}=bad \ \land\ Z_{it}=good \}$. We can argue that we can assume that $X_{it} \sim iid(\mu,\sigma^2)$ across $t$, since there's no reason why the time between a bad and a good call should influence the next one. Across the callers $i$ however, each can have different calling reasons and thus require more/less urgent responses leading to different $\sigma_i^2$. If so, then we could do the test for each class of callers. For simplicity let's assume the same behavior across all callers. Then, assuming by the CLT that the mean $\bar{\mathcal{X}}\overset{a}{\sim}N(\mu,s^2/n)$, where $n$ is the number of elements in $\mathcal{X}$, we could build a test statistic:

$$ t=\frac{\bar{\mathcal{X}}-\mu}{s/\sqrt{n}}\sim N(0,1) $$

where $s$ is the sample standard deviation. Now a sensible test to the problem at hand would be if the value of 40 is too stringent, i.e. classifies bad when a reclassification would be needed. The null would be $H_0: \mu \ge 40$. If we reject, then the actual mean time in which bad calls turn good is smaller than the threshold of 40, meaning that no chance is given to bad calls that quite likely would turn good. In that case, the number 40 should be replaced with a certain small percentile of $X_{it}$ from the data, i.e. the percentage of misclassifications that are tolerated. If we don't reject then the true mean is actually greater or equal to 40, meaning that the company gives bad calls if anything too much of a chance, when statistically they would keep being bad for a longer period.

Real-life approximation

The problem in real life is that we don't know the true $Z_{it}$ for all $t$, since if $Z_{it-1}$ is bad and the corresponding $X_{it} \le 40$, it automatically gets classified as bad too.

Your boss suggested/requested to calculate the time (with an upper limit of 1 week) between all calls from the same phone number that turned from bad to good within a period of 4 months. This is essentially an approximation for the set $\mathcal X$, with the assumption that every call that turned from bad to good within 4 months could have turned from bad to good at any time prior to that. For example: caller 1 calls at time 1,2,3 and 4. At time 1 he gets labeled as bad, at time 4 as good. By using the difference between times 1 and 2, times 2 and 3, times 3 and 4, one implicitly makes the assumption that the relabeling could have happened between any of those times - not only as it is shown, namely between times 3 and 4. I don't see the logic of calculating combinations of time differences (e.g. between times 1 and 3), since a relabeling can only happen between two consecutive calls - even more so, then taking the difference between non-consecutive times would wrongly increase the mean. Note that the time has to be calculated between calls of the same caller - i.e. it wouldn't make any sense to calculate the time e.g. between $Z_{11}$ and $Z_{21}$, since the second could call at 1 second before the first, which obviously doesn't tell anything about the bad to good switch.

Regarding the time period of 4 months, you need to specify a time where to get the data from, might as well do. The 1-week cap for the time between calls doesn't make much sense to me, unless there are very few callers that have a week in between their calls - otherwise it would wrongly decrease the mean.

The approach of trying to approximate $\mathcal X$ is correct. The restrictions should be as few as possible, so I agree that the filters are very arbitrary - might make sense again, only if very few cases (outliers) fall outside of those.

If we leave all the restrictions out, the model seems quite ok given the data at hand.

(Another restriction that would maybe make sense is restricting the set $\mathcal X$ only to callers who had at least 2 calls within the 40-hour period. Then the assumption made in the beginning (that the switch from bad to good could have happened at any time prior) becomes very realistic. All calls that have more than 40h between them should be disregarded. This might change the underlying question being asked though and hence maybe even the test would have to be adapted. [Haven't given it much thought yet.])

---

Regarding your questions: Questions 1 & 2 have implicitly been answered I hope. Regarding 3, you can bootstrap to calculate the standard deviance for the mean, but if we assume (asymptotic) normality and homoskedasticity it can be calculated by formula (see the test statistic). The CLT would be visible by taking many samples of increasing sizes each time and plotting the distribution. In your case you only have 1 sample. 4. Yes, you can check for normality of $X_{it}$ within $\mathcal X$, using different tests. You could also just argue asymptotically under the iid assumption, so you would need to argue that iid is realistic.