So it has been a little while since I've taken my last statistics course, and I wanted to double check that I am not making any kind of grave errors in my expected value calculations.

Quick bit of background, my data is consumer receivable data, each row containing customer name, method of payment (card/cash/check), amount of total, number of items purchased, amount paid of total, amount written off, balance, and some other variables that don't matter for this problem.

My data consists of about 4 million rows of data, meaning it won't all fit in Excel. This is why I have been using R to deal with this data. My task is to provide more accurate values as opposed to a completely generalized one. The person who previously held my position calculated the average percent of the bill paid and assumed that was the percent we could collect on all new purchases. This seems a little.... wrong to me.

My plan is to break down the data into subsections (there are several distinct types of products that we sell), then calculate the average percent of collected of those that are already collected on. Once doing so, I will calculate the percent of cases that were collected on, and multiply those together for an expected value. My question (I guess it's a question), is what am I doing wrong here? I'm sure there is something wrong.

I vaguely remember needing to multiply each case by the probability of it occurring for a completely accurate expected value, however my boss wants this project done relatively quickly and unless its a significant difference, I'll probably just use the averages.

Thank you everyone!


1 Answer 1


For the basic question of what an expected value (EV) is and how to calculate it, try Wikipedia. Sometimes calculating an EV will involve multiplication by probabilities and sometimes it won't, which roughly corresponds, respectively, to the case of computing a population EV (that is, the EV of a theoretical probability distribution) and the case of computing a sample EV (better known as the sample mean, since "(arithmetic) mean" and "expected value" are synonymous). You're computing a sample mean, so you probably won't be multiplying by any probabilities.

As for your practical problem, it sounds like you want to predict how much a customer will actually pay, given an invoice you've sent them. The old method used by your predecessor is simplistic, but it isn't necessarily the wrong way to do it; the real test of a predictive model is its accuracy as measured with a test set or cross-validation or a similar technique. There are all kinds of ways you could build a more sophisticated model, including the idea you mentioned of separating products into categories, and it's likely that some more sophisticated model will achieve better accuracy than the old method. This is one of the core concerns of statistics, so if you want to learn more, you could start with most any applied statistics textbook. Linear regression and model validation are what you want to focus on for this kind of problem.

Since you've said you need this done relatively quickly, however, learning basic statistics may not be an option. You could hire a consultant (shameless plug: like me!), keep on using the old method, or just tell your boss that data analysis is something that requires time and expertise to do competently enough that the results are better than no data analysis at all (which your boss may not want to hear).

  • $\begingroup$ The key question is whether the prediction will be based on the same (percentage) mix of cases as the existing data. If yes, then a single overall sample average will do. If not, it will be more accurate to calculate the sample average per case, then multiply by fraction of customers of that case for the prediction "scenario" and sum across cases to get the expected value for the prediction. $\endgroup$ Jun 16, 2016 at 22:44
  • $\begingroup$ @MarkL.Stone I would assume that if you're going to consider category information in training, you'll make category-specific predictions, too, rather than making a category-netural prediction with a correction for the difference in category distributions between the old and new samples. $\endgroup$ Jun 16, 2016 at 23:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.