0
$\begingroup$

I have a data set which contains avg profit and the number of samples for each data point, no more information.

I'd like to compare the data points and decide which one to focus on more. However, I'm not sure how to take the sample size into account. As a simplified example, I have written the python code below.

import matplotlib.pyplot as plt; plt.rcdefaults()
import numpy as np

objects = ('A', 'B', 'C', 'D')
y_pos = np.arange(len(objects))
sample_sizes = [10,5,20,15]
sample_avgProfit = [12,14,2,4]
weighted_mean = 0
for i in range (len(sample_sizes)):
    weighted_mean += sample_sizes[i]/sum(sample_sizes)*sample_avgProfit[i]

weighted_proportion = []
weighted_sum = sum([a*b for (a,b) in zip(sample_sizes,sample_avgProfit)])
for i in range (len(sample_sizes)):
    weighted_proportion.append(sample_sizes[i]*sample_avgProfit[i]/weighted_sum)


plt.bar(y_pos, sample_avgProfit, align='center', alpha=0.5)
plt.xticks(y_pos, objects)
plt.title('avg profit')
plt.show()

plt.bar(y_pos, weighted_proportion, align='center', alpha=0.5)
plt.xticks(y_pos, objects)
plt.title('weighted proportion')
plt.show()

You'll see that B has a higher avg profit, but since its sample size is small, when I consider weighted sum and average, then A shows a higher proportion weight (BTW, is this the right term for the values I calculated?)

So my question is:

  1. Am I using the right metric to compare the data points?
  2. How to interpret the results? in this example, product A might have a higher weighted proportion, but still B has a higher avg price. What is the right way to make a decision in this case?
$\endgroup$
0
$\begingroup$

Comparing means requires information about the variability of the estimates of the means. That is, in order to test hypotheses of the type "is the mean in group B higher than that of group A?" you'd need to know something about the standard deviation in each group. If one group varies wildly, then the mean is likely unstable too and should not be "trusted" as much as if the variance was very small. Without such information, you'd have to make heroic assumptions about what the standard deviations in each group is.

To me, the main question here would then be if the difference in the two groups' observed means is of practical importance to you? If the smaller group's mean is only slightly larger than the much larger group's mean, and the larger group's mean is good enough, then perhaps going with the group where you have a more precise estimate of its mean is the safer bet. But again, this is based on the assumption that the standard deviation in the group that you have more observations from is not so much larger group that it counteracts the larger sample size.

$\endgroup$
2
  • $\begingroup$ Thanks Phil. totally understand, so would you suggest a threshold on the sample size? grouping best means above the threshold in one group (say safe group) and best means below the threshold in another (risky group). then take different stategies for each of these group. Can you think of a better approach? $\endgroup$ – towi_parallelism Oct 25 '18 at 13:54
  • $\begingroup$ I don't know enough about the problem to answer that properly, I am afraid. However, I would also want to caution against regression to the mean, which is also always something to bear in mind. $\endgroup$ – Phil Oct 25 '18 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.