5
$\begingroup$

I have a dataset with observations which violate the independent and identically distributed (iid) assumption. The dataset is composed by a number of post-click conversion events (purchases on a number of websites), which have been attributed to one of two creatives on test. The attribution model consists on a last-click approach. In other words, each conversion has been assigned to the last clicked creative. In addition to that, the conversion window has been shrinked to only 12 hours before the conversion event. This means that to be eligible to be considered in our analysis, the conversion has to be preceeded by a click on an advert within the 12 previous hours.

As I wrote, the observations are not iid if we agree that it's not just the last advert you saw that affects your behaviour and make you click on that same impression (another more technical way to call an advert in the industry) and then buy something.

I was thinking to approach the problem in the following way, but I would like to have a feedback on how statistally sound this is, and if there are better alternatives. I must say, there is no way I can run a proper experiment at the moment, so I need to get the best out of what I already have.

I'm interested on testing if creative A drives a higher CTR (click-through rate) and AOV (average order value) than the other creative on test.

When analysing CTR, I would collect all data relative to the impressions that have been shown for both options:

  1. Treat each observation as a binary event (0: no click, 1: click)
  2. Sample 10% of the observations from each sample
  3. Test if both distributions resemble a normal distribution
  4. If the distributions resemble a normal shape, run a z-test (there are lots of observations, more than a 100,000) with $H_{0}: \mu_{A} - \mu_{B} = 0$ and $H_{A}: \mu_{A} - \mu_{B} \neq 0$
  5. If the distributions don't pass the normality assumption, then use a Whitney-Mann test to assess the null hypothesis of $H_{0}: \mu_{A} - \mu_{B} = 0$ and $H_{A}: \mu_{A} - \mu_{B} \neq 0$

When dealing with the the average order value:

  1. Consider as observations the conversion events and transform on a log10 basis the order value or each observation (order value will be asymmetric otherwise, considering the price of an item can't be negative in our case)
  2. Attribute each conversion to either creative A or B based on the last-clicked advert within 12 hours before of the conversion event
  3. Sample from each of the two sets only 10% of the observations (on OpenIntro Statistics this is suggested as a methods to use when we are not certain about the independence assumption)
  4. Use bootstrap to build, for each sample, the expected log10(order value)
  5. Verify, using a Whitney-Mann U test, if the boostrapped estimate of the log10(order value) of sample A is greater than the one of sample B

I've read bootstrap on certain circumstances can overcome the iid assumption, but I'm not sure if this can be the case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.