I am struggling to develop a sensible strategy or protocol for the predictive modeling of a complex set of data. Apologies in advance for the indeterminate nature of some of this description but it’s necessary to maintain confidentiality.

The data consists of a sample of nearly 1 million records representing a multilevel aggregation of daily consumer behavior across the last 4 years of time. The dependent variable is a single activity like sales dollars (not unit sales) spanning the entire competitive frame for this behavior. The activity is seasonal with spikes in the first of the year, later in the spring and again in the fall. The unit of analysis is similar to a product sku (stockkeeping unit) -- as would be found in retail clothing or insurance claims -- that have additional features attached to them. These “skus” are a granular subset of several higher layers of aggregation, e.g., skus within models within brands within business units. The features are similarly complex in structure, e.g., wide variations in pricing, hard physical attributes, softer differences in style or color, and so on. Each product is involved in marketing activities and the extent to which these products are able to capture this marketing activity is one endogenous measure of its success. The last week of activity for each sku will be used as a holdout or test set.

There are tens of thousands of skus, hundreds of categorical features, 100 brands and 30 business units. The skus themselves represent a single, massively categorical feature. ETL* is complete as most of the data is from a structured RDBMS (relational database mgmt system). The meaning of the features can and does vary as each business unit is free to define and use their features as they see fit (or understand them). The naming of the skus also varies and can even change from season to season within a business unit. Some of the skus represent used spin-offs or acquisitions of skus originating in another company. In addition, some of the features are unstructured and open-ended text, e.g., names of people. Normalization of the naming of the skus and features is not feasible.

The panel is unbalanced with about 30% of the data being right censored, another 30% is left censored and 10% are both. I’m comfortable handling the censoring with dummy variables and don’t want to integrate a survival function into the model. In addition, the activity ranges from the intermittent with potential gaps in time of several months (particularly for the low frequency skus) up to a continuous daily stream across all 4 years. Unit roots, autocorrelation and co-integration abound but given the brief time span of a large number of skus, time series methods such as VAR (vector autoregression) will not work. Diffusion processes are also evident as some of the skus can and do go “viral” after introduction, becoming blockbuster products.

Up to this point, most of this is manageable, possessing a reasonable line of sight into flexible, multilevel mixture model approaches to prediction. In addition, the processes of variable selection and dimension reduction are well understood since, out of all of this potential information, significant exploratory work has been done and a viable subset of features has been identified. For now, I don’t have access to a parallel platform and am doing the work on a single CPU (chip or computer processing unit). Slowly iterative, “divide and conquer” or random forest-type methods are one solution to this problem.

The thing that really has me sweating is the extreme value, heavy tailed nature of the dependent variable. Across all skus, it has a tail exponent of -1.15, which describes a Frechet or inverse Weibull distribution. Approximately 5% of the skus capture 20% of the activity, rippling across the competitive frame when they occur, sucking the air, so to speak, out of the day in doing so. This means that outlier identification and deletion does not make sense as significant amounts of activity would be tossed out. This also means that parametric, panel-based approaches to predictive modeling can be ruled out. It's also worth noting that I prefer not to transform, beyond a natural log, the dependent variable into something more tractable. Thus retaining its original scale.

Given all of this, what might be some feasible approaches to predictive modeling of this large set of massively categorical, heavy tailed data? Any thoughts? Suggestions? (Clarifications?)

**** Definitions ****

ETL is taking data from one system (extract), modifying it (transform) and loading it into another system (load). And not necessarily in that order.

  • $\begingroup$ The question is kind of unanswerable, because there can be one correct answer. Without seeing the data and getting feel, there are too much variables to account for. I can propose first of all doing individual forecasts and see whether the accuracy is acceptable, and only then proceed in building a bigger model. Also try forecasting the aggregated data, it should help with accuracy. A general advice is to start from simple models and then complicate things gradually. Set a concrete test of model goodness of fit to compare the models, to see whether there is a progress or not. $\endgroup$
    – mpiktas
    Apr 5, 2016 at 5:53
  • $\begingroup$ In general I would not worry about the the distributional properties of dependent variable. They might be explained by independent variables. If they are not, then it is a time for worry. $\endgroup$
    – mpiktas
    Apr 5, 2016 at 5:56
  • $\begingroup$ @mpiktas Many thanks for giving my statement thought! I agree that it's "unanswerable," but that's with the benefit of hindsight. Your suggestions are good ones. The only point I would make is that the question was an outcome of having worked with the data quite extensively over an extended period of time. This work pretty much covered the waterfront as comprehensively as I was able. Wrt your last comment, the extreme nature of the DV was not well explained by the IVs for a mean predicting model. Hence, the move to robust quantile regression. $\endgroup$ Apr 5, 2016 at 15:37
  • $\begingroup$ Thank you also for the literature suggested in your earlier comments responding to my other, more recent question. This will be of great use to me in further refining the work, bolstering my understanding of the issues sounded there. Cheers! $\endgroup$ Apr 5, 2016 at 15:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.