I was trying to do classification for census-income (KDD Data Set) (https://archive.ics.uci.edu/ml/datasets/Census-Income+%28KDD%29). The aim is to classify people who gain income >50 k and <50k. For certain attributes, which is capital losses, capital gains, dividends from stocks, its value is too different(idk how to explain it but below is the describe.() function for each attributes).
count 37382.000000
mean 1707.598657
std 10038.318677
min 0.000000
25% 0.000000
50% 0.000000
75% 0.000000
max 99999.000000
Name: capital gains, dtype: float64
count 37382.000000
mean 85.043818
std 411.696338
min 0.000000
25% 0.000000
50% 0.000000
75% 0.000000
max 4608.000000
Name: capital losses, dtype: float64
count 37382.000000
mean 571.174522
std 4122.600219
min 0.000000
25% 0.000000
50% 0.000000
75% 0.000000
max 99999.000000
Name: dividends from stocks, dtype: float64
There is so many 0 values and i dont know what should i do to even out the data spread. If i want to do binning, each features has about 117-2000 different values and it is not efficient to bin it one by one manually. Below is the value_counts() funtion.
capital gains
0 34187
15024 698
7688 384
99999 347
7298 270
10520 98
27828 94
20051 82
14084 77
5178 74
3103 67
13550 64
4787 63
10605 51
5013 40
9386 34
4386 33
8614 25
25236 23
3325 23
4650 21
7430 21
2829 20
4064 18
25124 18
4934 18
15831 16
6514 15
18481 14
6097 14
...
6767 3
1173 3
1455 3
1831 3
3456 3
914 3
3273 3
4931 2
3418 2
2463 2
3432 2
2009 2
2977 2
5060 2
2036 2
2290 2
7978 2
3887 1
6723 1
2538 1
6497 1
5721 1
6360 1
401 1
2936 1
2774 1
6612 1
2346 1
2227 1
8530 1
Name: capital gains, Length: 117, dtype: int64
capital losses
0 35790
1977 328
1902 249
1887 171
2415 107
1564 60
2258 54
2559 38
1602 36
1848 29
2444 27
2824 27
2377 27
1485 25
1408 18
2174 17
1876 17
1590 17
2001 16
1672 14
1719 14
2339 14
2392 12
1740 11
3004 11
2547 10
1974 10
2704 9
1579 8
1669 8
...
3175 2
2201 2
1380 2
1944 2
1870 2
1138 2
2206 2
1640 2
2603 2
1651 2
4608 1
419 1
1911 1
2179 1
2457 1
1573 1
2754 1
1539 1
2597 1
2352 1
1421 1
1648 1
880 1
1735 1
3770 1
1429 1
810 1
2267 1
1816 1
974 1
Name: capital losses, Length: 96, dtype: int64
dividends from stocks
0 29851
100 425
500 401
1000 390
200 305
50 280
2000 234
150 210
250 204
300 193
2500 176
1500 169
400 161
1 152
5000 148
25 121
3000 120
600 107
10000 98
4000 95
20 80
10 71
125 68
75 61
750 60
2 59
7500 57
5 53
6000 52
30 51
...
9181 1
48366 1
3850 1
43000 1
186 1
154 1
8286 1
204 1
332 1
12650 1
1849 1
428 1
3896 1
6671 1
556 1
1785 1
4750 1
780 1
3800 1
1625 1
5531 1
7087 1
1401 1
1164 1
1516 1
1337 1
1900 1
1932 1
6126 1
10171 1
Name: dividends from stocks, Length: 821, dtype: int64
wage per hour
0 26078
500 734
600 546
700 534
800 507
1000 386
425 376
900 336
550 280
1200 256
1100 235
650 229
450 222
1500 221
750 202
1300 198
850 167
525 147
1600 136
1400 132
1800 127
400 125
1700 116
2000 108
475 105
435 98
625 91
950 84
1050 76
575 73
...
521 1
1116 1
1394 1
892 1
828 1
1330 1
1266 1
796 1
764 1
1234 1
793 1
857 1
1033 1
604 1
2429 1
1129 1
8600 1
1074 1
1289 1
1353 1
978 1
914 1
1865 1
1929 1
1993 1
2155 1
170 1
234 1
458 1
1395 1
Name: wage per hour, Length: 1240, dtype: int64
Looking by percentile, the data is also very skewed. What should i do if i want to include this feature in my classification model? Or is it better to drop?