My boss has a PhD in Math from a top 30 university; point being he's a bright guy but doesn't have a formal background in statistics. For background, I have an MS in Statistics but focused more on machine learning. He asked me to do a hypothesis test but it's unlike anything I've ever seen before so I'm wondering if it even makes sense. We are both data scientists.
Presently, our company has a system that automatically classifies phone numbers and by extension, their calls, as "bad" or "good." When the first call by a phone number is made through our system, we undertake an expensive manual investigation of various attributes related to the phone number to determine whether it is a "bad" or "good" phone number. The relevant aspect of this system is that there is a business rule in place where a previously classified "bad" phone call all subsequent calls labeled as "bad," if those calls are made within, let's say, 40 hours of the first call.
The stakeholders admit this 40 hour threshold is somewhat arbitrary and we've been tasked with finding a new threshold, supported by data. My boss said I should do this:
Build a dataset using call data from the past 4 months. Each observation is a phone call. A single phone number can make multiple phone calls. This dataset should only contain numbers that started off the 4 month period classified as "bad" and was later classified as "good" at any point afterward, in the same 4 month period.
Calculate the time difference between ANY two calls made within the same week. A week is defined as a time period starting on Monday 12am and ending 11:59pm on Sunday. Clearly, this restriction makes it such that the max "time gap" between any two calls is 168 hours. For example, phone number 1 making 3 phone calls will result in 3 rows of data that looks like:
phone_number time_gap phone number 1 hours between call 1 & 2 phone number 1 hours between call 2 & 3 phone number 1 hours between call 1 & 3
3a. Create a new dataset that is a filter of the above data set such that
time_gap is in [20, 50].
3b. Create a new dataset that is a filter of the above data set such that
time_gap is in [20, 70].
Find the mean of datasets 3a, 3b. Calculate the Z-score for this test statistic, using the formula:
Z = (Xbar - mu) / (StD / sqrt(n))
Xbar = mean of a sample with replacement from 3a, 3b mu = 40 hours, the current business rule threshold StD = the standard deviation of a sample with replacement from 3a, 3b n = size of the sample from 3a, 3b
My questions are:
Does this even make sense to do? In my mind, by restricting the data in steps 3a and 3b, we are kind of manufacturing the results for our sample mean and just trying to show that calls in this range are statistically significantly different from 40. Then, so what? This seems like cherry-picking our data and seems to definitely break the rule of: "don't throw away data you don't like just because you don't like it."
This whole thing doesn't make sense to me because we're just comparing two arbitrarily defined subpopulations of the "bad to good phone number" population. Doesn't it make more sense to find the mean number of hours between any two calls, between 2 different populations? For example, the "bad to good phone number population" and the "bad to bad phone number population"?
Does it make sense to take repeated samples from datasets 3a and 3b, i.e. bootstrapping to get confidence intervals for my mean? My boss saw that I did that and laughed, saying that it's pointless because doing so is basically just trying to prove the Central Limit Theorem.
Don't I need to check for normality of the underlying distribution of the time gap field?
Detailed responses are appreciated. I've already made the points above to my boss on a few occasions but he basically said I just don't understand basic statistics (which is possible, lol) and has become pretty exasperated at this point. Normally, I'd just let it go and do the work without much argument but we (read: I am) are going to have to present and defend these results to external statisticians and I'm honestly not a believer.