I am struggling to develop a sensible strategy or protocol for the predictive modeling of a complex set of data. Apologies in advance for the indeterminate nature of some of this description but it’s necessary to maintain confidentiality.
The data consists of a sample of nearly 1 million records representing a multilevel aggregation of daily consumer behavior across the last 4 years of time. The dependent variable is a single activity like sales dollars (not unit sales) spanning the entire competitive frame for this behavior. The activity is seasonal with spikes in the first of the year, later in the spring and again in the fall. The unit of analysis is similar to a product sku (stockkeeping unit) -- as would be found in retail clothing or insurance claims -- that have additional features attached to them. These “skus” are a granular subset of several higher layers of aggregation, e.g., skus within models within brands within business units. The features are similarly complex in structure, e.g., wide variations in pricing, hard physical attributes, softer differences in style or color, and so on. Each product is involved in marketing activities and the extent to which these products are able to capture this marketing activity is one endogenous measure of its success. The last week of activity for each sku will be used as a holdout or test set.
There are tens of thousands of skus, hundreds of categorical features, 100 brands and 30 business units. The skus themselves represent a single, massively categorical feature. ETL* is complete as most of the data is from a structured RDBMS (relational database mgmt system). The meaning of the features can and does vary as each business unit is free to define and use their features as they see fit (or understand them). The naming of the skus also varies and can even change from season to season within a business unit. Some of the skus represent used spin-offs or acquisitions of skus originating in another company. In addition, some of the features are unstructured and open-ended text, e.g., names of people. Normalization of the naming of the skus and features is not feasible.
The panel is unbalanced with about 30% of the data being right censored, another 30% is left censored and 10% are both. I’m comfortable handling the censoring with dummy variables and don’t want to integrate a survival function into the model. In addition, the activity ranges from the intermittent with potential gaps in time of several months (particularly for the low frequency skus) up to a continuous daily stream across all 4 years. Unit roots, autocorrelation and co-integration abound but given the brief time span of a large number of skus, time series methods such as VAR (vector autoregression) will not work. Diffusion processes are also evident as some of the skus can and do go “viral” after introduction, becoming blockbuster products.
Up to this point, most of this is manageable, possessing a reasonable line of sight into flexible, multilevel mixture model approaches to prediction. In addition, the processes of variable selection and dimension reduction are well understood since, out of all of this potential information, significant exploratory work has been done and a viable subset of features has been identified. For now, I don’t have access to a parallel platform and am doing the work on a single CPU (chip or computer processing unit). Slowly iterative, “divide and conquer” or random forest-type methods are one solution to this problem.
The thing that really has me sweating is the extreme value, heavy tailed nature of the dependent variable. Across all skus, it has a tail exponent of -1.15, which describes a Frechet or inverse Weibull distribution. Approximately 5% of the skus capture 20% of the activity, rippling across the competitive frame when they occur, sucking the air, so to speak, out of the day in doing so. This means that outlier identification and deletion does not make sense as significant amounts of activity would be tossed out. This also means that parametric, panel-based approaches to predictive modeling can be ruled out. It's also worth noting that I prefer not to transform, beyond a natural log, the dependent variable into something more tractable. Thus retaining its original scale.
Given all of this, what might be some feasible approaches to predictive modeling of this large set of massively categorical, heavy tailed data? Any thoughts? Suggestions? (Clarifications?)
**** Definitions ****
ETL is taking data from one system (extract), modifying it (transform) and loading it into another system (load). And not necessarily in that order.
