What do you call this technique?

I'm a worker in the energy business, and I'm trying to see if we can predict whether a customer will complain given that they called in about x, y and z. My team and I are new to data science.

In the data there are about $$200$$ call reasons.

The way we tried to do this before was to construct a sparse matrix which tells you how many times customer $$i$$ calls in about $$j$$. The final column is whether $$i$$ complained in 2015. Sadly it appears that more than 30% of customers who've called in have complained. We tried random forests (70% accuracy), naive bayes (54% accuracy) and JRip (70% accuracy). All of these were cross-validated. Previously some people tried this task out and made a random forest model with 85% OOB accuracy. I don't know how they managed that.

Anyway I've been thinking about a different approach. The aim is to estimate the probability $$\operatorname p(s)$$ that a customer whose call reasons form the bag $$s$$ will complain. Pseudocode:

$$\text{let } h \text{ be a hash table where the indices are bags and the entries are 2-vectors}\\ \text{for row }r \text{ in data}:\\ \text{ }h[\text{the bag of call reasons in }r][r.\text{didComplain}] \text{ += } 1$$

Then $$\operatorname{p}(s)\approx {h[s][1] \over h[s][0] + h[s][1]}$$

After discussion with someone we reasoned that if we found someone who didn't complain with call reasons $$s$$ we should increment $$h[s'][0]$$ for every subbag $$s'$$ of $$s$$. If on the other hand they did complain we would increment $$h[s'][1]$$ where $$s'$$ is every superbag of $$s$$. I've decided to omit the pseudocode.

What do you call this? Is it a good idea?

1 Answer

If I understand your (pseudo)-code right, you are counting number of calls mentioning one specific call reason, and number of them which are classified as a complaint. So what you are calculating is simply called the relative frequency of complaints. It can be seen as an estimator of the probability of (this specific) complaint.

You can see the number of complaints as a random variable with a binomial distribution, and based on that you can calculate a confidence interval for the probability of complaint.