2
$\begingroup$

I am recently working on LASSO for my commercial car usage data. A problem I am facing now is that, I have around 200 factors, 100000 samples. However, quite a few of the factors are very sparse: only 1000-5000 has a value (all the rest are zero) among all these 100000 samples.

I am wondering if there is a good way to deal with this kind of sparse factor.

Can someone give me some insights here?

Thanks a lot!

$\endgroup$
1
  • $\begingroup$ there are some interesting things to say here. In fact, one of my classmates is working on a problem under this regime. I'll write an answer sometime in the future when I know more! $\endgroup$
    – user795305
    Commented Jun 27, 2017 at 16:15

1 Answer 1

5
$\begingroup$

If by "sparse", you mean that the factor value is unknown for 95% of your rows, then there is not much you can do besides not including that factor in your regression.

If by "sparse", you mean the value of that factor is known to be 0 for 95% of your rows, then there is no problem with just including the factor normally in the regression. Of course, it might still be that a linear dependency on that factor is a poor model, or that its coefficient is not significantly different from zero, but there is no problem per se with regressing against factor which is different from zero only a small fraction of the time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.