Im just reading the wunderful book Probabilistic Programming and Bayesian Methods for Hackers. In the very first chapter they inspect whether the habits of one single customer change over time, for example: We count the number of text messages received by a single user:

enter image description here

and they model the situation as follows: the $x_i$ (the numbers we actually observed) are outcomes of a random variable $X = (X_1, ..., X_n)$, i.e. $x = X(\omega)$ and that $$ X_i \sim \text{Poisson}(\lambda_i) $$ where for a (potential) switchpoint $\tau$, $$ \lambda_i = \begin{cases} \lambda_1 & \text{if}~ i \leq \tau \\ \lambda_2 & \text{else} \end{cases} $$

($\lambda_1, \lambda_2$ and $\tau$ are drawn from respective random variables $\Lambda_1, \Lambda_2$ and $T$ and we know their distributions).

Then using some Monte Carlo methods one can apply Bayesian inference in order to approximize the posterior densities $f_{\Lambda_1|X}$, $f_{\Lambda_2|X}$ and $f_{T|X}$. For example, $f_{T|X}$ looks like enter image description here indicating that there is a switchpoint around day 45.

Now let us say I am some company making money with transporting messages. Recently I started some Campaign (Advertisements) and I want to see whether or not the campaign was worth its money. So I'll need to define some success measurements. Informally, a good choice would be to 'count' the number of messages, the people received (or sent) before and after the start of the campaign and if that number increases 'suddently' at one switchpoint (the day the campaign was started or a little bit later) then I'll be willing to say that it was worth its money.

So I want to proceed as above but now I'm facing the problem that for every day $i$, I have, say, millions of users sending/receiving messages. The next logical thing I came up with in order to fit this into the framework as above is to count the average amount of messages received or sent during a fixed day. However, I see two problems here:

Lets say $U = \{u_1, ..., u_N\}$ is the set of users and user $u_j$ received $m_{i,j}$ messages on day $i$.

1) If for a fixed day $i$, I compute $$m_i := \frac{1}{N} \sum_{j=1}^N m_{i,j}$$ then although [up to identifying the range $\mathbb{N}_0/N$ with $\mathbb{N}_0$ again] the $m_i$ are Poisson distributed again (correct?), it does not feel right to treat them like this (because this switch from $\mathbb{N}_0/N$ back to $\mathbb{N}_0$ does feel unnatural).

2) I would like to argue (using the central limit theorem) that the $m_i$ are normally distributed so that I could apply the same Bayes machinery as above to normally distributed random variables with unknown (=to be determined) $\mu$ and $\sigma$. However, the problem here is that I do (a) not know that all the users have the same parameter in their Poisson distribution and (b) know that the $m_{i,j}$ (in $j$) depend on each other: If a user $j_1$ is a good friend of user $j_2$ then their numbers of received messages influence each other. Can I still say that there are only a 'few' of them depending on each other?

How to proceed?

I neither have a concrete task nor concrete data so any pointers/hints/... are appreciated :-)




I don't see the need for making the switchpoint a random variable in this case, as you know when the treatment is happening. You could instead treat this as a kind of paired T-test, with each observed data point being the post-switchpoint average of each user minus the pre-switchpoint average of the user (these windows you will need to determine).

The distribution of these changes will have millions of observed values. You could do something like Krushke's BEST version of the T-Test, but I would just look at that distribution of differences and see if it's centered around 0. You may want to be on the lookout for a mixture distribution as it could imply there was an effect on a subgroup.

Ideally, treatment would be randomized and you could compare against the change of non-treated, but that may not be possible.

I like the switchpoint method as well, but I'm just not convinced it's the right tool for this problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.