1
$\begingroup$

My data consist of time series of Wikipedia hits for football players. I want to use the CausalImpact package to explore the effect of the World Cup on their popularity which I base on their Wikipedia page hits. This is effectively seeing if the tournament works as a marketing campaign for their popularity on Wikipedia.

I use the Wikipedia hits from players who featured in the previous tournament as a baseline and exclude the players who featured in both as that would confound things.

date       Messi  Ronaldo # 2018 players
01/01/18     756    871      
02/01/18     660    743
03/01/18     523    453
04/01/18     412    654
05/01/18     488    568
06/01/18     515    345

date        Lahm    Xavi # 2014 players
01/01/18     253    171      
02/01/18     164    443
03/01/18     323    351
04/01/18     312    654
05/01/18     183    263
06/01/18     325    145

In R, I loop over each one of the time series for the players in this World Cup and compare them against a predictor series which is all of the players at the previous tournament.

But this seems a bit unwieldy because the time series are so variable. Some players barely get any hits, others have marked seasonality and others are constant. What is the appropriate way to select the predictor time series in a case like this? I've seen this answer but it refers to datasets being contemporaneous.

List = NULL
    for (i in 1:length(levels(wikiHitsWC2018$article))){
      data <- zoo(cbind(data_wideWC2018[i], data_wideWC2014[,1:119]),time.points)
      impact <- CausalImpact(data, pre.period, post.period,model.args = 
list(nseasons = 7, season.duration = 1))
      List[[i]] <- impact
    }

Hope you can help!

$\endgroup$
  • $\begingroup$ I am not sure how you 'compare them against a predictor series' but if it is by mean root square of the distance (or similar) between the player and the predictor series then this is a very crude way to assess what differences are going on. Instead of quantifying, I believe you can better start exploring your data with some graphs (to see what you actually need to quantify), and after that, when you have a good idea of the patterns, test for the distribution of the parameters in the models for those patterns. Possibly the descriptive statistics already convey a story about the differences. $\endgroup$ – Martijn Weterings Aug 25 '18 at 9:41
2
$\begingroup$

A few thoughts:

  • Level of your data: In general, the level of your data determines the level of your inference. bsts followed by Causalimpact is a time series approach to impact estimation suitable for cases where one just doesn't have individual level data. In a B2B setting for example, one often only observes sales over time and the intervention date, e.g. marketing campaign. This is not the easiest data to work with and certainly not to make causal inference. In this time series setting, bsts and Causalimpact nevertheless provide a strategy to get an estimate of the intervention impact.

    Now in your case, you have much more granular data, i.e. repeated observations of individuals over a long time frame. This is a balanced panel and somewhat the best data you can get, since in many ways, panel data can enrich your analysis and allows you to do things that time series data can't! Your data thus allows you to use regression discontinuity, panel data models, propensity score matching, or bsts & CausalImpact with matched comparison group. I outline below how this could look like.

  • Control group: A suitable comparison group in your case would be players who pre-qualified. This is a good comparison group since the pre-qualified eligible players match core team players very closely in skill, past performance and other characteristics relevant to the outcome you study: popularity of a player. Things that you need to account for in this set up are 1) non-random assignment of treatment, 2) spillover effects on the comparison and 3) pre-existing differences between treatment and comparison group.

(1) Selection into treatment occurs since you have to be really good to play in the core team at the world cup, which is not very random. Clearly, players get select into treatment (i.e. the core team) based on observable measures of skill and past performance, but likely also unobservable ability and motivation.

As a result (3), treatment and comparison group characteristics of players will differ pre-treatment and thus won't be balanced between treatment and comparison as it is the case under random assignment.

(2) A key assumption in any treatment comparison model is that the treatment has no effect on comparison players. However, a generally higher interest in football will have some effect on eligible players too, so the world cup has spillover effects on the comparison. A partially treated comparison group, in turn means the treatment estimate will be biased downward. Controlling for these problems requires understanding more about the data generation process and usually means collecting more granular data like yours.


  • RDD: A suitable identification strategy in this set up would be regression discontinuity design (RDD): For this you would need a score that measures player quality. Possibly this is publicly available. If not, you could construct a score that separates eligible players and core team players by some threshold, thus reverse-engineering the selection criteria to be admitted as a core team player in the world cup. This could be a weighted average of e.g. past performance and other continuous skill measures. A probit model explaining treatment on the past performance and skills would not work here, as it does not allow for discontinuities. In any case, it should be credible to say that selection into the treatment happens based on some threshold in this score, which is the cut-off that is exploited in regression discontinuity design (RDD).

    Essentially, you assume that players very close to the cut-off are similar to an extent that some of the differences between them are random. This is realistic since for example some player may have had an injury, thus less training time and then performed less well that season. This can happen to any player. Hence within some narrow range around the cut-off, the characteristics of players are similar and the threshold can be thought of as a natural local experiment where the differences between the core and eligible players are "as good as random". If you now pick N core and eligible players near the cut-off in your quality score, you can then evaluate the effect of the world cup on core player popularity by plotting popularity (e.g. #clicks wikipedia or googletrends series hits) on the vertical axis and quality score for all players on the horizontal. To test whether the world cup increases popularity of players you basically look for a discontinous jump in popularity around the cut-off point in player quality. If so, you can attribute the jump to the fact that the player was a core player in the world cup, rather than just an eligible player. The estimated treatment effect is the size of the jump and if you also want the significance of the jump you can use local linear regression with a subset of the data around the threshold or polynomial regression with all the data. In either cases, your assumptions are that the model is linear and that treatment is independent of the X, i.e. that you managed to remove all selection on observables.

  • RDD works if you have a score that explains treatment assignment. You also need enough individuals within some narrow band around the cut-off. Further, it deals with problems 1-3. The score takes account of the non-random assignment (problem 1), by converting it to a local randomised experiment around the cut-off, which explains the selection into treatment. Problem 2 is about the key assumption that comparison players are Not affected by the treatment. Since the world cup has a non-exclusive publicity effect on any football player. Hence spillover effects on the comparison occur, which may bias the treatment effect downward. If you think these spill-over effects are significant, the causal effect of your RDD estimate will be a lower bound. As for problem 3, you have turned it into a local random experiment, where no pre-existing differences should exist between eligible players that marginally failed to be selected and the core team players.


  • To use bsts & CausalImpact on your data, you focus on the time series dimension of your data: Hence, you could predict individual core player popularity based on individual or aggregate popularity of eligible players. If your variation is high, maybe reducing the frequency of the data to a weekly level helps. Equally, you could take aggregate popularity of eligible players to predict aggregate popularity of core team players around the treatment as suggested before. Also, if you feel you don't have enough data with wikipedia series, maybe googletrends series for the player name is less sparse? In any case, it's worth looking at it, since you can use it in the regression component.

  • Selection of regression components: The validity of bsts & Causalimpact depends on the validity of covariates, which should a) mimic the trend of the target in the pre-treatment as closely as possible and b) be relevant to the outcome you are studying. The first part is a prediction task, so you practically, you could start by plotting the potential time series and choose the most correlated ones as a pre-selection. You can then apply standard model selection techniques, i.e. select the optimal number of regressors based on how well you could fit the target in the pre-treatment period. (one step ahead out of sample error, Harvey statistic). Part b) needs an answer in the context of your data, which in your case means you need to make model adjustments and choose covariates that account for problems 1-3!

  • To account for this you could use matching methods, which aim to balance characteristics between core team players and eligible players such that that the only remaining difference between the two groups is treatment assignment. This is done based on selected variables that are relevant to the outcome you are studying, i.e. popularity. One option is to use propensity score matching, which explains the treatment selection process based on observable characteristics. Effectively, it is like doing instrumental variables on the treatment, which is endogenous here and you end up constructing a score that measures the probability of being in the treatment group. You can then pick the eligible players that most closely match the core team players using Kernel density matching for example. At this stage, you have tried to control for the non-random assignment and selection into treatment in a similar way as you did under the RDD by balancing characteristics around a score. This accounts for problem 1 and 3, as long as there is no remaining unobservable differences between treatment and comparison group. Spillover effects will still downward bias your impact estimates. You can accept the estimate as a lower bound of the actual effect or try to estimate the size of the spillover effect separately.

Hence, the analysis could look as follows:

1) Match eligible players (comparison) with core team players (treatment) based on observable pre-treatment characteristics, which you can scrape from e.g. Wikipedia. The output of this step defines the comparison group.

2) Run bsts & CausalImpact using the comparison identified in the previous step.

If you want to predict individual-level effects, you may find this useful.


  • Now the only issue that you have not tested for is selection on unobservables which may confound the true relationship between a cause and an effect and thus significantly bias estimate results. The key advantage of panel data is that it optimally exploits and combines the time series and cross-section data generation process of your data and thereby allows you to answer questions that just cannot be answered with time series data alone, i.e. how to control for unobserved common factors. If for example, selection into the core team was in a rush under time pressure or includes family background of the player or some kind of foul play, this will result in selection based on unobservables. Since it's unobservable, it's harder to control for, but several options exist and you can test for it too.

  • Control function approach: If you have reason to believe that the selection on unobservables is a problem, you can test for it by comparing the estimates of a model that controls for sample selection versus a model w/o controlling for selection. If the coefficients are stable across the two specifications, then this is evidence that any selection bias is likely to be small; if the coefficients move substantially, then this indicates that selection bias significantly affects estimated results. If the latter is the case, you can control for selection using a control function approach similar to the two-step approach developed by Heckman. First, you explain treatment assignment using relevant regressors and construct the inverse mills ratio from the predicted score. Then you run a regression that explains popularity based on the regressors, including the Mills ratio.

Some more thoughts:

  • Choice of outcome variable: If you want to measure the effect of publicity on a single player, maybe $\frac{n\_wikipedia\_page\_hits\_player\_X}{n\_wikipedia\_page\_hits\_whole\_team}$ would be a metric that better reflects the effect on the player normalised by the popularity of the team.

  • Choosing an appropriate start for the pre-/post period needs careful consideration: Clearly, the treatment is somewhere between when the team constellation was released (the public learnt about it started looking at the players' profiles) and the date of the last game in the tournament (player publicity reached its peak).

    • To measure the effect of releasing the team constellation, setting the start of the evaluation at the constellation release date may be suitable, especially if you are interested in the pure publicity effect of the tournament, independent of the player's performance. Also, in the case that this "release effect" is strong, setting the start date for impact evaluation later, can make your effect negative easily, since you are measuring the impact of a series that gets back to its normal level after a temporary shock.

    • Setting the start of the treatment at the date of the tournament would measure the effect of the incremental popularity of a player due to their participation, performance at the games and other news/scandals at the time of games with regards to the player and the team. The latter is more than a marketing effect you are measuring.

$\endgroup$
0
$\begingroup$

Given the high variability of the page hit data, there is probably not much you can do to fit models for individual players. However, you could at least try to fit a model for the sum of the daily page hits for all 2018 players (and also use the sum of the page hits for the 2014 players to have a covariate with a similar variability).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.