We have an existing study of media consumption which is online and offline.

Online study: To measure consumption online, we use log data from web accesses and various other sources. We have reliable socio-demographic data for a big quantity of those online sessions (such as sex, location, household incomes and similar stuff).

Offline study: To measure offline consumption, we use panels (face to face interviews, telephone interviews, and similar techniques). We gather the same socio-demographic data as the online study. The sample (panel) is 8000 persons. In the interview, they do mention if they consumed a given content media offline or online.

Is there any technique that allow reliable data merging given the fact most persons in the sample are not measured online or there is no way to know if they are?


The quick answer is "yes" but with caveats. There are actually more than a few considerations to be made in merging data from disparate sources. These challenges require a few simplifying assumptions to enable meaningful analysis. Regardless, any data matrix resulting from this merger will be necessarily complex, noisy and unbalanced with inherently large amounts of missing information and measurement error. If you want to think in terms of a class of models, my view is that this structure would fall into pooled, cross sectional, panel data models or, more specific to your situation, marketing mix models. Wooldridge's Econometric Analysis of Cross Section and Panel Data is one of the best books on econometric theory. Lee Cooper's Market Share Analysis is one of the best applied books out there on marketing mix models. In addition, there are many papers that address specific aspects and concerns regarding these models -- a few of these will be discussed below. Regardless most, if not all, solutions will eliminate the possibility of doing household or individual level models. In other words, any resulting solution will be relatable and projectable only to broadly defined groupings of information.

The first and simplest solution is to bind the data sources together based on a common temporal unit such as day, week or month. One of the first papers to discuss this approach was written by Moore and Winer and titled A Panel-Based Method for Merging Joint Space and Market Response Function Estimation.(1) They propose a basic framework enabling merger or fusion of information from disparate sources. That said, algorithmic and probabilistic data fusion of widely disparate information is a topic with a pretty large literature behind it. Just do a google search for "data fusion methods."

In general, the overall lack of control with web information introduces many complications, particularly wrt classic, theoretic, econometric models of panel data which assume neat, completely observed and balanced data inputs. In your case, the first assumption would have to be that the samples (web vs survey) contain independent information since, as you state, there is no way of knowing whether a survey panelist was online or not. Even probabilistic matching algorithms won't work with information as indeterminate as this.(2)

Next, consideration should be given to how the data is rolled up or aggregated by time and place. Weblog information can be quite temporally granular (down to the minute or second), unbalanced wrt sample size as well as temporally irregular since visitors' migration onto the site is not controllable. Are these logs for a single site or across multiple websites? There are decisions to be made as to whether the data is aggregated to the level of the web page, the website, etc., as well as by location or geography. How is the uncertainty wrt web panel membership and assignment to be handled, e.g., is it to be assumed away? Are cookies used to identify the same visitor across multiple trips to the site? Given that demographics are available for many of these visits, do you create demographically cross-classified cohorts, such as by age, sex, location, etc, over time and aggregate that way? How do you handle visits that are missing demographics? What's the best temporal unit for this aggregation? The choice should be a function of how web activity is to be matched to the available timing of survey responses. How far back does it go? SME or domain knowledge is best here but priority should be given to aggregating web data up to something close to the, almost certainly, coarser temporal unit of the survey data. Is this aggregation in terms of counts of activity? Are these counts to be cumulated? Does the site activity have multiple types of possible behaviors? Is it useful to incorporate activity by "type?" Finally, if marketing activity has been used in an effort at driving site traffic and revenues, then that information should be included in the model as a control or adjustment to parameter estimation. (This does not mean you should be required to evaluate the effectiveness of these instruments as a by-product of the analysis). Without additional information from the OP, one has no way of knowing what an appropriate aggregation might be.

Survey-based information, on the other hand, is much more controllable as respondents are likely drawn from a national probability sample of some kind, producing demographically balanced data wrt both time, place and sample size. That said, how granular (or projectable) is the information by geography? National? Regional? State level? With 8,000 respondents, it's not likely to disaggregate much lower than state level. Is the survey based on rolling, continuous sampling with a given number of respondents per day or week? Is it quarterly? Annual? How far back does it go? For instance, if you only have one annual sample of 8,000 respondents, that's a very important piece of information to know as it makes much of the previous discussion moot. With only a single survey data point, web activity should be aggregated within the bounds of that sampling frame. With continuously sampled data, there can be considerable volatility in measurement even at an aggregated monthly level. Given this, one approach to stabilizing this would be to create rolling panels of information. This would both smooth the information and give it a more robust sample size while also introducing, by definition, autocorrelated residuals due to the rolling structure. Care would need to be given to how this is handled statistically. Given the panel data framework, the standard Box-Jenkins, ARIMA-type approaches and fixes for autocorrelated, univariate time series would not be appropriate. To the best of my knowledge (other readers of this question should push back here), there aren't any good, widely agreed on solutions for messy, unbalanced, noisy panel data. Huang and Fitzmaurice's paper Analysis of Longitudinal Data Unbalanced over Time comes as close to an answer to this issue as anything I'm aware of in the literature.(3) Another good reference is to the spatial econometrics literature which considers geo-spatial autocorrelation structures as a by-product of their models.

Having discussed temporal aggregation, it is not a rigid requirement that the two differing domains of information be in the same unit of time. For instance, Ghysels (at Duke) has developed a model for mixed frequency data (MIDAS for MIxed DAta Samples) where, e.g., the features or predictors can be in one unit while the target or dependent variable is in another, usually higher, unit of time.

Depending on how disaggregate you choose to make the matrix of information, the results may be sparsely populated in spots, e.g., by geography and/or time. Sparse data models are a separate and, for the most part, new class of applied models with a literature unto itself.

Then there are considerations wrt how the inevitable missing values are handled. In an earlier CV post the issues concerned with model-based multiple imputation of missing data was given some attention (here ... A data set with missing values in multiple variables). Another approach would be to forget about using so-called "single source" information that requires construction of a fully populated matrix for all variables and features. The workaround here is to revert to a massive covariance matrix that is kind of like a big, uncontrolled fractional factorial experimental design where the missing information is pooled across the covariance framework and structure. This approach would result in incomplete and inconsistent information across the full matrix but would enable aggregate, directional inference. This suggestion would also most likely involve the application of random matrix theory to massive covariance structures. However and to the best of my knowledge, it has not been developed, much less well developed, in the literature. You would kind of be on your own in pursuing it.

These are just a few suggestions wrt the questions that need to be addressed. Once the data has been cleaned and aggregated, the OP is ready to move onto the grittier concerns related to exploring and modeling the results. As always, one can't emphasize enough the need for kicking the exploratory can of the data. As already suggested, panel data model methods would be a strong contender for an overall modeling framework. One issue to watch out for is that there can be significant convergence problems with models based on data like this. Depending on how massive your information is (not so much in terms of size but in terms of the sheer number of features), you may want to revert to the various iterative machine learning, approximating approaches broadly known as "divide and conquer," "split and conquer," or even "bags of little jacknifes," which are all variants and extensions of bootstrapping, jacknifing and Breiman's random forest method. Good references for these include Chen and Xie's paper A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data,(4) as well as Wang, et al's, paper A Survey of Statistical Methods and Computing for Big Data.(5)

In my view, all of these fairly subjective choices, which for the most part lack any cookbook-type benchmarks, conventions or guiding rules of thumb, will result in a model producing, at best, approximate results. This means that there will be little in the way of "ground truth" against which to benchmark the results. However, as Kaiser Fung has noted in his book NumberSense(6), models lacking a benchmark "ground truth" can evolve into self-benchmarking systems as the predictions from them prove either useful (i.e., profitable) or not.

* References *

1) Moore and Winer, A Panel-Based Method for Merging Joint Space and Market Response Function Estimation, Marketing Science, 1987, vol. 6, no. 1, pps. 25-42.

2) William Winkler, Machine Learning, Information Retrieval, and Record Linkage, ungated copy at https://www.niss.org/sites/default/files/winkler.pdf.

3) Huang and Fitzmaurice, Analysis of Longitudinal Data Unbalanced over Time, Journal of the Royal Statistical Society. Series B (Statistical Methodology), Vol. 67, No. 1 (2005), pp. 135-155.

4) Chen and Xie, A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data, Statistica Sinica 24 (2014), 1655-1684, http://www3.stat.sinica.edu.tw/sstest/oldpdf/A24n49.pdf.

5) Wang, Chen, Schifano, Wu, and Yan, A Survey of Statistical Methods and Computing for Big Data, http://www.math.chalmers.se/Stat/Grundutb/GU/MSA220/S16/ReviewBigDataR.pdf.

6) Kaiser Fung, Numbersense: How to use big data to your advantage, McGraw-Hill, 2013.

  • 1
    $\begingroup$ Thanks for the great answer! I'll read all the references and will maybe come back here. Unfortunately, I'm not allowed to upvote it :( $\endgroup$
    – Sharp
    Mar 11 '17 at 17:38
  • $\begingroup$ Np re upvote. Let me know if you have any questions. I'll do my best to try and answer them. $\endgroup$ Mar 11 '17 at 18:26
  • $\begingroup$ I figured out that I can just accept the question to let you get your reward :) $\endgroup$
    – Sharp
    Mar 13 '17 at 6:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.