# Automatic feature building/extraction

I have a large time stamped data set (several millions of rows), with known measured inputs xi, where i is a large number to the order of magnitude of 20. The goal is to predict a response yi given the inputs.

Now the question deals with building and testing features or functional forms that best encode the relationship between y and x. Are there techniques/algorithms that automatically do it for you, in other words test out various forms of features combining different x's in various functional forms, where the algorithm is intelligent enough to suggest to you which functional forms made more sense to nudge you in the right direction?

I know that building good features is an art, I guess what I am asking is if there are brute force techniques to assist me in that process.

Thanks, please let me know if I was unclear.

• The Recurrent Neural Networks are good when it comes learning features in sequence data AND when there is lots of training data. – Vladislavs Dovgalecs Feb 5 '16 at 18:46

In my opinion the answer to your question is alternatively called a Dynamic Regression/Transfer Function/Polynomial Distributed Lag/Autoregessive Distrbuted Lag/XARMAX. The whole idea is form a minimally sufficent model of the form (shown here with just 1 X where you have 20) using as few lags as necessary. Note that sometimes B (the backshift operator) is replaced by L , particularly in the "dismal science" of econometrics.

Yt=μ+(ω0−ω1B1−.....−ωsBs)/(1−δ1B1−...δrBr)Xt−b+et


which easily restates to :

an XARMAX model with different subscript notation

Y[t] = a[1]Y[t-1] + ... + a[p]Y[t-p]
+ w[0]X[t-0] + ... + w[r]X[t-r]
+ b[1]a[t-1] + ... + b[q]a[t-q]
+ constant

The statistical problem is to determine what the appropriate lag structure is for each Y and X.

The statistical problem is to create a robust solution that incorporates any needed Pulses/Level Shifts/Seasonal Pulses/Local Time Trends that exist in the data and are not treated by any of the user-specified X's.

The statistical problem is to validate that the final model's parameters are invariant over time and the error variance is free of any non-constancy . Non-constancy in the error process can arise in both a stochastic and non-stochastic manner.

In terms of commercially available solutions , I can recommend AUTOBOX as a potential http://www.autobox.com/cms/ as I have helped develop it. Additionally other players in this environment are SPSS and SAS to name a few.

Care should be taken when selecting an approach that the data should not just be used BUT challenged as to it being part of the process and not reflecting unusual activity thus data-cleansing procedures need to be in place and effective.

• Thanks for the reply, I am not looking to learn linear relationships necessarily. I guess my question is how to learn sets of functions zj = hj(x) that could be used an an effective features while predicting y? – gbh. Feb 21 '15 at 19:26
• That's what Automatic Model Building/Identification/Forecasting does. It is a productivity aid that doesn't (necessarily) need you to know anything. If you do have knowledge then the process might be improved. – IrishStat Feb 21 '15 at 19:30