Describing Specialization

Question

I am trying to formalize an observed trend. To simplify what I am trying to do, suppose a dataset of salesmen selling items A, B, C, D. In the year 1950, the proportion of each sold is (0.2, 0.2, 0.1, 0.5), however this shifts to (0.4, 0.1, 0.1, 0.4) in 2000 (or some other statistically significant shift) - now this can be shown with a chi-square test. Important to note, the group of salesmen in 1950 is not the same as in 2000.

Now what I would like to show is that taking into account the shift in product sales, we witness a specialization in certain products, ie whereas in 1950 salesmen would sell all products fairly equally, in 2000, salesmen increasingly focus on certain products (for example, whereas in 1950 an average salesperson may have a distribution of sales more or less representative of the sales overall - 0.2, 0.2, 0.1, 0.5; a 2000 salesperson may have 0.9 of A and 0.03 of each of the others).

I was wondering how one would go about this? Would it be appropriate to compare the top 10% of sales people in each product and show increasing discrepency in how they sell when compared to a normalized supposed salesperson? Is there a more standard way of doing this?

Any help would be very much appreciated.

Answer 1

Perhaps you might want to read up on diversity indices. Perhaps you've heard of the Gini index, which quantifies income inequality, or what economists know as the Hirschman-Herfindahl index to quantify market concentration (the concept appears to have been discovered first by Edward Simpson, and it's called the Simpson index in ecology). A higher Herfindahl index means more market concentration, i.e. you have one firm with most of the market share.

Or, for those of us who are familiar with latent class analysis, many of us have heard of (Shannon) entropy, which we use to describe how well-separated are the latent classes.

Example for individuals as the unit of observation

I'll give an example using normalized Shannon entropy (note: link to free article at the American Psychological Association) because I'm most familiar with it. Entropy (not normalized!) for each unit of analysis (e.g. each salesperson, or each metropolitan area, etc) is given by the formula:

$E = -\sum^C_{i=1}p_i \ln p_i$

Above, $C$ indexes the number of categories of items (or latent classes, racial groups, etc). Assume that $\ln 0 = 0$.

Imagine Mrs. Chen, a very specialized salesperson, sells only item D, i.e. (0, 0, 0, 1). Her entropy is 0 under this calculation.

Now, imagine Mrs. Huang, who sells all items in equal proportion, i.e. (0.25, 0.25, 0.25, 0.25). Her value of entropy is $-4 \times 0.25 \times \ln 0.25 = 1.3863\ $, i.e. she has the maximum possible value of entropy given that you have 4 types of items to sell. You might want to normalize entropy by dividing by the maximum possible value of entropy, which is $\ln C$. Here, $\ln C = \ln 4 = 1.3863$, so Mrs. Huang's normalized entropy is 1.

Example for groups of observations or samples

In latent class analysis, we would normally calculate the normalized entropy over all observations,

$E = 1 + \frac{1}{N \ln C}\sum^N\sum^C_{i=1}p_i \ln p_i$

(Note: this is from the first formula in the link above, with notation modified to be consistent with the rest of the answer)

So, the above formula should tell you how much the entire salesforce is specialized in each year. Remember, if all individual salespeople have the exact 1950 proportion of sales, then you have one value of entropy, but you could have a situation where 50% of the salesforce was selling only product D, 20% were selling only A, etc. That would still be a pretty specialized salesforce, and you'd see that in the entropy value.

As illustrated in Budescu and Budescu (first link), the Simpson/Herfindahl index, which they call generalized variance (GV), should perform equivalently to entropy. The calculation is a bit simpler, but either should be easy enough to do. If you're in Stata, install the entropyetc package from SSC (Nick Cox, who contributes here frequently, is the author). There have to be R packages that do this also, but I can't be bothered to search for a specific one.

Answer 2

You are working under the assumption that every salesperson should sell every item uniformly, so what you want to find out is how discrepant those observations are under your assumption or what was the probability of seeing such discrepant observations.

If you want to know is how discrepant those observations are with the prior you have, then I would measure the Kullback-Leibler Divergence from those 2 distributions.

If you want to know how likely a distribution so discrepant with your observation was, then I would do a Multinomial Test.

There are other statistical tests that do the two things at once, like Kolmogorov-Smirnov Test, but they work under the assumption that the distribution is continuous, I don't think this works for your case.

Answer 3

Interesting question.

Taking only the top 10% of sales people in each product would mean you could only draw inferences about a particular subset of your population, so I don't think that would be a good idea unless you're mostly interested in that subset of salespeople.

As a really simple approach to test if sales have become more specialised: for each salesperson in your dataset, you could work out the proportion of sales from their best-selling product (e.g. in 2000, salesperson 1 sells mostly product A, at proportion 0.9 of their sales). Then compare these proportions between 1950 and 2000, e.g. using a t-test with a logit transformation on your proportions before you carry out the t-test (assuming that you just have data from these two years - otherwise using regression with time as your explanatory variable).