0
$\begingroup$

I have a long list of access logs, associated with some persons, lets say web access-log. I want to build prediction model for some target varaible, associated with persons, and i have a training set for this.

What troubles me, is that to prepare training set, containing some extraction from web logs, like domain name + number of pages visited, i need to build some flat table with enormous number of columns, representing unique domains (10k or maybe 100k depending on part of the log i will extract).

What is a common approach for such problem? Should i try to reduce dimentionality first, trying co group up domain names? But that will be a loss of data, because i believe some combination of domains can influence target variable. Or should i learn some algorithms, able to work with "longitudal" data, because most techniques i familiar with require data to be flat?

Thanks for any advice and direction, my primary instruments are R and SQL

Improve this question
$\endgroup$
5
  • $\begingroup$ random sample features, such as random forest is a good start. $\endgroup$
    – Haitao Du
    Commented Jul 12, 2016 at 19:40
  • $\begingroup$ But to start random forest i need to provide flat table with features, right?) $\endgroup$
    – Ema Nymton
    Commented Jul 12, 2016 at 19:42
  • $\begingroup$ some trick here feature hashing. scikit-learn.org/stable/modules/… $\endgroup$
    – Haitao Du
    Commented Jul 12, 2016 at 20:02
  • $\begingroup$ I have a log to transform, lets say 3-5 millions of records, which, transfromed will get me 100k columns (my estimate) and like 100k rows $\endgroup$
    – Ema Nymton
    Commented Jul 12, 2016 at 20:07
  • $\begingroup$ If I am doing this, I will sample first. Then investigate the benefit of adding rows and columns. In many cases, double the data(rows) have little benefit. (I am talking about if you have a "high bias model") $\endgroup$
    – Haitao Du
    Commented Jul 12, 2016 at 20:16

2 Answers 2

Reset to default
0
$\begingroup$

Loss of data isn't all bad in statistics. Consider the simple average: useful in many instances because it throws a lot of the data away. Anyhow, if I'm not mistaken, using PCA might well be a very effective solution in this situation. Give it a shot.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But to do PCA i have to do the job of making large flat table first, right? $\endgroup$
    – Ema Nymton
    Commented Jul 12, 2016 at 18:05
  • $\begingroup$ The way you're describing things, I would think so. $\endgroup$
    – readyready15728
    Commented Jul 12, 2016 at 18:21
0
$\begingroup$

There are couple of approaches you can test:

  1. Hypothesis based features: if you have a list of errors which you know are linked to response variable you will create features
  2. Frequency based features: If you are building ML model very low occurring features might not be impact if response variable is not associated to it.
  3. I am not a fan of PCA analysis but I generally use regularised regression algorithm like lasso to reduce the dimension. If the numbers of variables is too much to handle I would do a uni-variant gini/ uni-variant analysis and reduce the variable set.

Hope it helps. Happy learning!!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.