0
$\begingroup$

I was trying to do classification for census-income (KDD Data Set) (https://archive.ics.uci.edu/ml/datasets/Census-Income+%28KDD%29). The aim is to classify people who gain income >50 k and <50k. For certain attributes, which is capital losses, capital gains, dividends from stocks, its value is too different(idk how to explain it but below is the describe.() function for each attributes).

count    37382.000000
mean      1707.598657
std      10038.318677
min          0.000000
25%          0.000000
50%          0.000000
75%          0.000000
max      99999.000000
Name: capital gains, dtype: float64

count    37382.000000
mean        85.043818
std        411.696338
min          0.000000
25%          0.000000
50%          0.000000
75%          0.000000
max       4608.000000
Name: capital losses, dtype: float64

count    37382.000000
mean       571.174522
std       4122.600219
min          0.000000
25%          0.000000
50%          0.000000
75%          0.000000
max      99999.000000
Name: dividends from stocks, dtype: float64

There is so many 0 values and i dont know what should i do to even out the data spread. If i want to do binning, each features has about 117-2000 different values and it is not efficient to bin it one by one manually. Below is the value_counts() funtion.

capital gains
0        34187
15024      698
7688       384
99999      347
7298       270
10520       98
27828       94
20051       82
14084       77
5178        74
3103        67
13550       64
4787        63
10605       51
5013        40
9386        34
4386        33
8614        25
25236       23
3325        23
4650        21
7430        21
2829        20
4064        18
25124       18
4934        18
15831       16
6514        15
18481       14
6097        14
         ...  
6767         3
1173         3
1455         3
1831         3
3456         3
914          3
3273         3
4931         2
3418         2
2463         2
3432         2
2009         2
2977         2
5060         2
2036         2
2290         2
7978         2
3887         1
6723         1
2538         1
6497         1
5721         1
6360         1
401          1
2936         1
2774         1
6612         1
2346         1
2227         1
8530         1
Name: capital gains, Length: 117, dtype: int64

capital losses
0       35790
1977      328
1902      249
1887      171
2415      107
1564       60
2258       54
2559       38
1602       36
1848       29
2444       27
2824       27
2377       27
1485       25
1408       18
2174       17
1876       17
1590       17
2001       16
1672       14
1719       14
2339       14
2392       12
1740       11
3004       11
2547       10
1974       10
2704        9
1579        8
1669        8
        ...  
3175        2
2201        2
1380        2
1944        2
1870        2
1138        2
2206        2
1640        2
2603        2
1651        2
4608        1
419         1
1911        1
2179        1
2457        1
1573        1
2754        1
1539        1
2597        1
2352        1
1421        1
1648        1
880         1
1735        1
3770        1
1429        1
810         1
2267        1
1816        1
974         1
Name: capital losses, Length: 96, dtype: int64

dividends from stocks
0        29851
100        425
500        401
1000       390
200        305
50         280
2000       234
150        210
250        204
300        193
2500       176
1500       169
400        161
1          152
5000       148
25         121
3000       120
600        107
10000       98
4000        95
20          80
10          71
125         68
75          61
750         60
2           59
7500        57
5           53
6000        52
30          51
         ...  
9181         1
48366        1
3850         1
43000        1
186          1
154          1
8286         1
204          1
332          1
12650        1
1849         1
428          1
3896         1
6671         1
556          1
1785         1
4750         1
780          1
3800         1
1625         1
5531         1
7087         1
1401         1
1164         1
1516         1
1337         1
1900         1
1932         1
6126         1
10171        1
Name: dividends from stocks, Length: 821, dtype: int64

wage per hour
0       26078
500       734
600       546
700       534
800       507
1000      386
425       376
900       336
550       280
1200      256
1100      235
650       229
450       222
1500      221
750       202
1300      198
850       167
525       147
1600      136
1400      132
1800      127
400       125
1700      116
2000      108
475       105
435        98
625        91
950        84
1050       76
575        73
        ...  
521         1
1116        1
1394        1
892         1
828         1
1330        1
1266        1
796         1
764         1
1234        1
793         1
857         1
1033        1
604         1
2429        1
1129        1
8600        1
1074        1
1289        1
1353        1
978         1
914         1
1865        1
1929        1
1993        1
2155        1
170         1
234         1
458         1
1395        1
Name: wage per hour, Length: 1240, dtype: int64

Looking by percentile, the data is also very skewed. What should i do if i want to include this feature in my classification model? Or is it better to drop?

$\endgroup$

1 Answer 1

0
$\begingroup$

There are several methods to deal with it:

  1. Add a constant value © to each value of variable then take a log transformation

    log(x+1) which has the neat feature that 0 maps to 0. log(x+c) where c is either estimated or set to be some very small positive value.

  2. Replace zero value with mean.

  3. Take square root instead of log for transformation

Hope this helps!

For more info u can refer this post: 'How should I transform non-negative data including zeros?'

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.