I work at an ad agency and one of our key performance metrics is what we call "cost per outcome". Right now I have advertisements grouped by type of advertisement, lets say type "A", "B", and "C" and have 500,100, and 70 advertisements respectively in each group and their associated cost per outcomes.

I need to classify these cost per outcomes into categories "low cost" "medium cost", "high cost" for each group so that I can make a comparison between the groups.

The approach I have currently taken to do this for each group is to simply classify any cost per outcome < 25th percentile as "low cost", >= 25th and < 75th percentile as "medium cost" and >= 75th as "high cost".

My question: is there a better approach that I can take for doing this type of estimation and classification or is this a sound method? What I am thinking is this approach is looking at a snapshot in time of these metrics and down the road the percentiles and classifications will probably change, but I don't know if my estimates should take that into account and don't really know how I would do that.

Please advise! Thank so much.

  • $\begingroup$ When you find 'percentiles' is that relative to ads in the same type A/B/C, or relative to all 670 ads? // Why three types A/B/C? Treat each type separately? Or is one goal to compare types? // You say one goal is to predict future performance.. I don't see how any classification as H/M/L now would help with prediction unless you know from historical studies that such H/M/L classifications are stable over time. It has been said, "Prediction is difficult--especially, about the future." (Attributed an old Danish saying, Danish physicist Niels Bohr, or American baseball player Yogi Berra.) $\endgroup$ – BruceET Jul 21 '20 at 18:34
  • $\begingroup$ Hey Bruce, thanks for the reply. Yes, I mean finding percentiles withih the same ad type. I want to compare the ranges of the low, medium, and high categories across the groups (A, B, C) as well as try to use those percentile ranges (or whatever is the most sound way to split the cost pers) as an estimation of what the "true" ranges are. $\endgroup$ – Keith Siopes Jul 21 '20 at 21:15
  • $\begingroup$ Well, determining H/M/L via quantiles for each type could make it different to compare groups using categories H/M/L. Will give example in Answer format. Not room in Comment format. $\endgroup$ – BruceET Jul 21 '20 at 21:27

Actual costs by ad type. Here are cost per outcome scores (in dollars, pesos, bitcoin, or whatever) for three hypothetical types of ads. I will explore them to see whether various kinds of possible summaries may be useful.

a = round(rnorm(500, 20, 3), 2)
     Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    10.66   17.86   19.79   19.84   21.89   29.60 
b = round(rnorm(100, 10, 2), 2)
     Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    4.680   8.777   9.890   9.973  11.165  14.900 
c = round(rnorm( 70, 30, 4), 2)
     Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    20.94   27.10   30.19   29.89   31.95   39.99 

For Type a ads, Medium cost is between 17.86 and 21.89, Low below 17.86 and High above 31.95; for Type b ads Medium is between 8.78 and 11.17; and for Type c ads Medium is between 27.10 and 31.95.

In the boxplots below (of varying widths as a reminder that sample sizes differ), the values inside the boxes correspond to Medium-cost ads.

boxplot(a,b,c, names=c("a","b","c"), varwidth=T,
        col="skyblue2", pch=20, main="Cost per Outcome")

enter image description here

Now, how would you make comparisons between Types? "Medium-cost" adds have very different cost ranges, depending on ad Type.

"Standard" costs. You might try standardizing the scores for each type separately. To do that, for each cost in a you would subtract the average cost for a, then divide by the standard deviation of costs for a. Similarly for b and c. You could call the standard costs $A, B$ and $C.$ They not longer represent actual costs, but relative costs within their ad Type. That might help make comparisons of ads among various Types, but if you start to talke about budgets, you'll have to go back to the actual costs $a, b,$ and $c.$

A = (a-mean(a))/sd(a)
      Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
  -2.87313 -0.61880 -0.01607  0.00000  0.64222  3.05702 
B = (b-mean(b))/sd(b)
      Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
  -2.62840 -0.59362 -0.04117  0.00000  0.59198  2.44674 
C = (c-mean(c))/sd(c)
      Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
  -2.18049 -0.67830  0.07431  0.00000  0.50211  2.46318 

boxplot(A,B,C, names=c("A","B","C"), varwidth=T,
        col="skyblue2", pch=20, main="Standard Cost per Outcome")

Now the lowest relative costs for Medium ads would be $-0.62, -0.59,$ and $=0.67$ for Types A, B, and C, respectively. They are nearly the same on the relative scale, but of very different actual costs. Again with relative costs within Types, the medium cost ads lie within the boxes of the respective boxplots.

enter image description here

Because I don't fully understand the differences among ad Types, the goal of your analysis of costs, and so on, I am not necessarily recommending you use any of the summaries or graphics above. But I'm hoping that thinking about these methods may help you clarify what you should do.

  • $\begingroup$ Thanks, Bruce, you're the man! I think the most important part of my question was whether or not using percentiles to create ranges of cost metrics which could be used to create the H/M/L groups was a sound method. In terms of making comparisons, I think perhaps finding a way of combining the standard deviation and the mean of the cost per metrics could perhaps be useful i.e. picture a grid which is split into quadrants by the mean of the stdevs and mean of the means of the cost pers so the lower left quadrant represents groups with low cost, low variance and upper right high cost, high var $\endgroup$ – Keith Siopes Jul 21 '20 at 22:55
  • $\begingroup$ the example I gave only included 3 ad groups, but there are really more like 50 groups $\endgroup$ – Keith Siopes Jul 21 '20 at 22:56
  • $\begingroup$ If your ad costs have right-skewed distributions then you will have higher variability for High-cost ads. // My formula for standard scores is commonly used and you might be able to find some information on how others have used them. But there are other ways of putting distributions on roughly the same scale. For example (subtract the type median, then divide by the type IQR). IQR ('interquartile range') is the distance btw the upper and lower quartiles. // Whatever you try, wishing you success. $\endgroup$ – BruceET Jul 21 '20 at 23:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.