1
$\begingroup$

I know that there have been similar questions but most of them have not worked, this is why I start this question.

I have a very large dataset (around 2,500,000 records) with approximately 100 variables, most of which are categorical and some of them numerical. There are a lot of missing values which are probably MCAR/MAR (the data come from an automated log file). Also, some categorical variables have a lot of different values (e.g. city attribute). I am trying to perform feature selection to find the most informative variables with regard to a discrete numerical target variable.

I have two problems:

i) How to fill in the missing values: most of the standard packages' functions get stuck (i.e. mice, hmisc). Should I use just the mode/median or fill in some default "unknown" value? If I decide to take a subset of the rows to decrease the size, how small should this subset be - if there are any empirical rules?

ii) Most of the feature selection techniques I have tried (fscaret, fselector, boruta) after using some simple imputation technique either hang or give me an error. I believe this has to do with the fact that some categorical variables have many possible values. What are some practical ways to deal with this?

Improve this question
$\endgroup$
9
  • $\begingroup$ What is the RAM available prior to loading the data into R and RAM available after loading? $\endgroup$
    – Sergey Bushmanov
    Commented Oct 12, 2015 at 18:22
  • $\begingroup$ Thank you, I will update once I have the number. It takes a while to load the data. $\endgroup$
    – user90772
    Commented Oct 13, 2015 at 17:32
  • $\begingroup$ If you suspect getting an error is a matter of imputation algo, you may try imputing on a fraction of data, say 1/10, but my suspicion you do not have enough RAM cause I estimate your data size is in Gb's $\endgroup$
    – Sergey Bushmanov
    Commented Oct 13, 2015 at 17:57
  • $\begingroup$ Thank you again, I have the same suspicion, but since I am a complete beginner, I guess I need to use some stratified sampling technique to get a representative sample, right? $\endgroup$
    – user90772
    Commented Oct 13, 2015 at 18:06
  • $\begingroup$ Keep it simple. Use stratified sampling, if you can, to make your sample representative for imputation techniques you apply. My point, again, your data set appears a little too big, at least for my laptop. On top of that, R is not very efficient in memory management. Make it sure your imputation algo works on a smaller sample, and then try increasing the size. $\endgroup$
    – Sergey Bushmanov
    Commented Oct 13, 2015 at 18:35

1 Answer 1

Reset to default
1
$\begingroup$

I had a similar problem. I used the $impute$ (from Hastie and Tibshirani) package that does K-Nearest Neighbor imputation. It was designed for micro-arrays, that means gene expression data which is continuous. But if you convert your predictors, all of them to numeric, it does a nice job and very fast.

source("bioconductor.org/biocLite.R") biocLite("impute")

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you Chamberlain, I will try this. I was wondering if you tried some feature selection on this dataset. $\endgroup$
    – user90772
    Commented Oct 13, 2015 at 8:45
  • $\begingroup$ Feature selection: the "LASSO" will be very suitable for this job. I am a big fan of Hastie and Tisbshirani. Use the $glmnet$ package. It will require patience. $\endgroup$
    – Chamberlain Mbah
    Commented Oct 13, 2015 at 9:00
  • 1
    $\begingroup$ @ChamberlainFoncha Think twice before using Lasso for feature selection, it wasn't made for this. Lasso uses regularization to induce some bias to ensure better out-of-sample generalization of the Lasso model, not other models. If you carry over features selected by Lasso to other models, there is absolutely no guarantee this set of features will perform well. $\endgroup$
    – Sergey Bushmanov
    Commented Oct 13, 2015 at 18:54
  • $\begingroup$ Feature selection with lasso is perfect. The parameter estimates are biased all in one direction. The selected set of predictors are the best. This subset is ordered that means if it is too small, reducing shrinkage will add more variables. The existing variables will remain in the set. You can then go ahead to do unbiased estimation with modeling without penalization on the subset if you are in for hypothesis testing. The lasso is perfect for feature selection. But if you need unbiased estimates, you already have the subset to work from. $\endgroup$
    – Chamberlain Mbah
    Commented Oct 13, 2015 at 21:39
  • $\begingroup$ If you are in for prediction then you can use the lasso both for feature selection and parameter estimation. In that case your biased estimates will then have a small out-sample variance. Please note feature selection is a goal in itself. $\endgroup$
    – Chamberlain Mbah
    Commented Oct 13, 2015 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.