0
$\begingroup$

I have about a million rows of data being collected every day, and I am trying to predict a government figure which is released on a less frequent basis, about once a month. Compared to a traditional ML training set where each example would have an actual value to use for training, I have much smaller training set.

I'm currently just aggregating my daily data to one "monthly" value, and trying to run a regression against the monthly government figure once it's released. Unfortunately, I only started collecting this data, so I don't even have enough data to run a simple regression model.

Is there any better way to do this, that would leverage the millions of rows of data I have? Despite there not being any 'correct answer' for each day of data due to the lower frequency of government data being published?

$\endgroup$
0
$\begingroup$

Have you thought about capturing sub-month temporal features of your data like volatility, spectrum, number of peaks, weekly trends, etc? There are many ways to capture aggregate information about temporal data, not just a simple average. The question of course is what makes sense i.e. describes your subject domain.

$\endgroup$
  • $\begingroup$ I have, but without any corresponding data on its correctness, I'm not sure if it would help. Thanks $\endgroup$ – skunkwerk Jan 1 '16 at 2:38
  • $\begingroup$ I am afraid I am not getting your line of thinking. What should be correct? The ultimate measure of a usefulness of a feature is whether it helps you to make a better prediction. You can do that only after you do create it and put into a model for evaluation. $\endgroup$ – Diego Jan 1 '16 at 5:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.