1
$\begingroup$

This is dataset that is going to be data mined for factors that affect an output that of interest

A large Part of my dataset (150 of 300 potential inputs) has a heavy skew of Zero values. usually this is >90% and always >70% zeros (true zeros)

This has risen as the input variables are usually OR options. -Say i have 10 input variables of the above in a wider dataset. of these 10 any one data line will have approximately 1-4 of these 10 populated with non zeros.

In real world terms if you use input variable 1 you may not need variable 2 to 10 the above statement isnt absolute. you may use a some of input 1 and some of 2 and none of 3 to 10.

i was tempted to transform these into binary catagorical 0 = No 1 = Yes but i would loose the scale on the non zeros. -If input variable 1 IS NOT 0 the values may range between 2 orders of magnitude

Improve this question
$\endgroup$
4
  • $\begingroup$ What is your question? $\endgroup$
    – whuber ♦
    Commented Jun 30, 2014 at 15:49
  • $\begingroup$ Sorry, i seem to have completely omitted that. Is there standard practice for this type of data? Do i need to transform or otherwise deal with this type of data in a particular way? Ie is categorizing the data a viable method? Or by doing this is the loss of the remaining 10% continuous information important? $\endgroup$
    – Taylor85
    Commented Jun 30, 2014 at 16:12
  • $\begingroup$ If you have 70% zeros your data are not continuous. If they're continuous apart from the 0s that would be a mixed distribution. $\endgroup$
    – Glen_b
    Commented Jun 30, 2014 at 23:29
  • $\begingroup$ yes this was the term i needed. It is a mixed distribution $\endgroup$
    – Taylor85
    Commented Jul 1, 2014 at 7:42

1 Answer 1

Reset to default
0
$\begingroup$

It is important to understand what you are going to do with the data. If you are focusing on predictive modeling, as you hint at, then it depends on the algorithm.

I would personally suggest using decision tree based algorithms (such as the venerable Random Forest) since they would be able to capture most of the intrinsic information in the raw data form. They will naturally segregate the zeroes if they behave distinctly from the non-zeroes for a given feature.

If you use an algorithm more sensitive to scaling (such as Neural Nets or SVMs) then you will want to be more careful about the skewing of the data. It might be worth a simple square-root transformation to reduce the chances of a feature having too much influence. I would definitely not do a standard scaling (i.e. subtract mean, divide by stdev) since that would cause some influential values. For the most part, you want the resulting coefficients to plausibly come from the same distribution.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I forgot to mention that often adding some ~meta variables can be helpful, especially if your goal is just prediction accuracy. An example might be, how many features are non-zero for this observation. If you have more knowledge of the context of the features this might make more-or-less sense. $\endgroup$
    – Shea Parkes
    Commented Jul 1, 2014 at 2:03
  • $\begingroup$ This is very useful and fills me with a bit of confidence. I felt trees would handle this data in its untouched form and was one of my initial options. The other option i was planning to use was a neural network. I often find when/how to deal with data like this tough $\endgroup$
    – Taylor85
    Commented Jul 1, 2014 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.