How to find the sweet spot

In R I have data where head(data) gives

day   new_users   promotion
1        33        20.8
2        23        17.1
3        19         1.6
4        37        20.8

Now day is simply the day (and is in order). promotion is the promotion-value for the day - it's simply the cost of advertisements on television. new_users is the number of new users we got that day.

In R I plot the data plot(data$promotion, data$new_users, col="darkgreen") and we get The plot indicate that we have a positive correlation, ie more promotion we get more new users. In R I test for positive correlation:

cor.test(data$promotion, data$new_users, method="kendall", alternative="greater")

which gives us a very low p-value, ie we have positive correlation.

Finding the sweet spot

I want to find a sweet spot, that is a point where the increase of promotion don't effect (or don't increase) new_users.

# Setting the promotion-value to 24
promotion_rate = 24
# Sub setting data so we only have promotion-value higher than 24
data_new =  subset(data, data$promotion > 24) # Testing for positive correlation cor.test(data_new$promotion, data_new$new_users, method="kendall", alternative="greater" ) I have done this for different values for promotion_rate. The results are for all promotion-values below 24 we get a low p-value, ie we have positive correlation in these cases. For promotion-values higher than 24 we get a p-value higher than 0.05, ie we do not have a positive correlation in these cases. Now is it valid to conclude that 24 is the sweet spot ? Update I have now plotted the cumulative sum of new_users - in R I type plot(cumsum(data$new_users), xlab="days", ylab="cumulative sum of new_users", col="darkred") Similar I plotted the cumulative sum for promotion. The blue is new_users and the orange is promotion.

plot(cumsum(data$new_users),xlab="days",col="blue") points(cumsum(data$promotion), col="darkorange") But this looks like a straight line so is it even possible to find a sweet spot?

• "promotion is the promotion-value for the day - it's simply the number of times an advertisement has been on television." — This description makes me think promotion should be an integer. How can an advertisement be on television 20.8 times? – Kodiologist Jun 13 '16 at 11:23
• It's the cost. I have edited it. – Ole Petersen Jun 13 '16 at 11:34
• How is this question different from yours here: datascience.stackexchange.com/questions/11915/…? – Dr_Be Jun 13 '16 at 12:35

By "sweet spot," I think we can assume you mean the inflection point -- the point where the growth in new users rolls over and begins to flatten out towards an asymptomtic max. There are no shortage of ways to analyze this information. One of them is as a diffusion process. Something that might help you visualize this would be not to treat it as a scatterplot but rather to plot the cumulative number of new users by day. The shape of that curve should suggest the inflection point. The basic idea is that growth is S-shaped -- slow at the beginning and the end with a rapid rise in the middle in the curve.

Mathematical modeling of that process began with Gompertz in the early 19th c but there are many other, newer models. This wiki post (https://en.wikipedia.org/wiki/Gompertz_function ) describes that model: Formula

{\displaystyle y(t)=a\mathrm {e} ^{-b\mathrm {e} ^{-ct}},} where

a is an asymptote, since {\displaystyle \lim _{t\to \infty }a\mathrm {e} ^{-b\mathrm {e} ^{-ct}}=a\mathrm {e} ^{0}=a} b, c are positive numbers b sets the displacement along the x-axis (translates the graph to the left or right) c sets the growth rate (y scaling) e is Euler's Number (e = 2.71828...).

(Apologies for any bad formatting)

In the marketing of new products, Rogers' diffusion model is one of the most widely cited papers in any field. His model was given mathematical formulation by Frank Bass and has seen many amendments and variations over the years.

Bass, F. M. (1969), “A New Product Growth Model for Consumer Durables,” Management Science, 215-227

Other models were developed in biological mathematics to describe the growth of, e.g., pea pods. Known as the Fisher-Pry transform which is described here (here). Fisher-Pry has been applied to the diffusion of new technology by groups such as the Program for the Human Environment at Rockefeller University.

All of the models mentioned so far basically involve univariate analysis. Extensions to multivariate regression models have been made recently. A good resource for those more advanced models (which would facilitate introducing promotion spend as a covariate and include R code) are available from these lecture notes:

http://www.unc.edu/courses/2008fall/ecol/563/001/docs/lectures/lecture27.htm

Here are the contents of that website:

• Overview of nonlinear mixed effects models
• Deciding which parameters should be made random in linear mixed effects models
• Centering a predictor to reduce parameter correlations in linear models
• The kestrel data set
• The Gompertz model
• selfStart functions in R
• Deciding which parameters should be made random in a Gompertz mixed effects model Interpreting the parameters of the SSgompertz function
• I just upvoted your answer, because it is good, even though it uses a completely different definition of 'sweet spot' than the original question. – Bernhard Jun 13 '16 at 12:38
• @Bernhard Thanks for the upvote. To your point, I don't agree with you and, even after rereading the post, think that my interpretation of the OPs "sweet spot" as an inflection point is consistent. Let's leave it up to the OP. – Mike Hunter Jun 13 '16 at 12:46
• Justifying my point: "I want to find a sweet spot, that is a point where the increase of promotion don't effect (or don't increase) new_users" is the point, where the curve has become flat. There is no room for an asymptotic max, whereas you describe "the point where the growth in new users rolls over and begins to flatten out towards an asymptomtic max". Obviously, when it begins to flatten, it is not yet flat and your definition has no problem with an asymptotic maximum. It would be advixable for the OP to take over your approach. – Bernhard Jun 13 '16 at 12:54
• Thanks for the response. "rather to plot the cumulative number of new users by day" - so I should plot the sum on y-axis and have trp (sorted trp) on the x-axis I assume? To compare the cumulative number against trp I need to have sorted trp. – Ole Petersen Jun 13 '16 at 13:21
• Actually, the y-axis is cumulative new users and the x-axis is the day, not trp – Mike Hunter Jun 13 '16 at 13:54

You cannot deduct from the data, that such a point really exists. You have a theory in your head, that at some Point more trp is not going to gain more users but that is not in your data. You will need to formulate this believe as a mathematical modell, then fit your data to that modell and then you can ask your question to the modell. For example you could believe, that an exponential function describes the relationship, then fit an exponential function to the data and investigate, when the slope of the exponential function gets so low, that you think it equals Zero for practical purposes. Or you might want to fit an polinomial curve and look for a place with true slope of zero. The p-value of correlation depends a lot on whether you have enough data Points in a particular Intervall.