I am looking to predict the outcome of a business process (inherently complex). I have four years of data across 15 companies.

I am trying to:
(1) understand key drivers in this process (what variables have strongest impact on outcomes
(2) look at these drivers across the samples to come up with a rough model

My questions:
(1) If I difference the data can I put them together into one seemingly long time series so I can employ lasso or random forest? In other words it's almost as if it were cross sectional/longitudinal combined...
(2) If yes to 1, the data is still possibly not stationary after taking the difference of the ln (p-value of .26 on adf test)? Is there another way to deal with this?

Any suggestions would be most appreciated!


1 Answer 1


You don't say what kind of data you have for these 15 companies beyond that it's cumulative. Is it sales data? Is it annual? Quarterly? Monthly? Your answers to these questions can help guide the approach you take.

First off, you can analyze the cumulants of your dependent variable (DV, whatever it is) using diffusion or logistic growth curve models. See the Program for the Human Environment website at Rockefeller University and search for publications related to logistic growth or "loglets." In the marketing literature this approach is called Bass-Anderson new product models. While this approach can give you useful qualitative insights such as growth rates, inflection points and likely asymptotic ceilings in terms of predictions, it doesn't permit much flexibility to integrate additional predictors that are relatable to your DV since it's driven almost entirely (some would say tautologically) by processes internal to the univariate time series (unless you leverage the extended Bass model). In addition, the approach completely ignores the primary concerns of the next technique, Box-Jenkins (B-J), regarding making the residuals HAC (see below for definition). In large part, this is due to the typically sparse data available from new product sales. Pooling the 15 cross-sections would be doable technically but challenging for a novice and you would probably end up having to build separate models for each company. Diffusion models can be most effective when there are limited amounts of data available for analysis. I don't recommend using them when reasonable quantities of time series information are to be had.

If you decide to analyze the differences vs the prior period, then there are two broad avenues: Box-Jenkins type models or, as you note, pooled time series.

Box-Jenkins models have many variations and aspire to take lags and moving averages (p's and q's in B-J terminology) to create white noise or HAC (heteroscedastic-autocorrelation consistent) residuals which have been detrended, deseasonalized and de-everything else. B-J is quite greedy with respect to the amount of information required, e.g., if you have 48 monthly data points to your time series that should be enough to initialize these models but if your data is quarterly, B-J becomes kind of problematic since the lags and mvas use up so many data points as to make a test and holdout breakout of the data impossible. Another challenge is that B-J techniques are univariate and you have 15 cross-sections. This means that, e.g., Dickey-Fuller tests for a unit root or co-integration across multivariate data like yours haven't been developed and you will be forced to build separate models for each company. To me, B-J is a wasteful, painful exercise when faced with large numbers of cross-sections and sparse data. A good overview of this class of models is Dominique Hanssens Market Response Models: Econometric and Time Series Analysis but there are many, many others.

Pooled time series (PTS) is a flexible modeling framework and as the literature documents, pooling facilitates "shrinkage" of the parameter estimates due to the increased degrees of freedom. One limitation (for some analysts) is that issues like autocorrelation are assumed away due to the fact that multivariate Durbin-Watson tests for serial correlation don't exist yet (like the D-F). The published workaround is to test the D-W 15 times and take an average, but that's kind of silly in my opinion. Simply put, PTS isn't focused on answering questions related to the autocorrelations internal to the univariate time series. Unless you choose to integrate a bunch of lags or mva's, pooled time series also has the advantage of being quite conservative with respect to the data required to initialize a model -- sparse data is no limitation (within reason, of course). In addition, you can add parameters for the companies (cross-sections), seasonality, trends, acceleration in the slope (the rate of change of the rate of change) and any other factor that's available to you, up to the point at which you would be overfitting the data (B-J enables this too with transfer functions). One important question is the extent to which your DV differs from one company to the next -- this can be in terms of the DV's metric (e.g., units vs revenues) or level differences in magnitude. Many analysts will transform the data (e.g., using a log normal or log-centering) to minimize these artifactual cross-sectional differences.

A great, highly readable introduction to PTS is Lee Cooper's Market Share Analysis available free on his UCLA website. Ignore the marketing and share aspects as his book covers the waterfront in terms of how to build and specify these models with great care given to functional forms to tease out differing types of elasticities and cross-elasticities, illustrations of data structure, and so on.

To your specific question re lasso or RFs, unless you generate dozens of candidate predictors, I don't see how they can be of much help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.