1
$\begingroup$

Hello I am working on machine learning model for count data, and I have various features that are highly skewed. The frequency table for one of the feature is given below.

0.0        142730
1.0         30338
2.0         11985
3.0          5727
4.0          3151
5.0          1907
6.0          1183
7.0           912
8.0           678
9.0           532
10.0          436
11.0          332
12.0          308
13.0          236
14.0          217
15.0          207
16.0          178
17.0          151
18.0          135
19.0          128
20.0          127
22.0          114
21.0          112
23.0           89
24.0           81
26.0           62
27.0           61
25.0           59
29.0           57
32.0           56
30.0           52
31.0           48
28.0           48
34.0           46
35.0           44
37.0           40
33.0           39
36.0           33
44.0           31
42.0           31
40.0           28
38.0           27
39.0           27
47.0           25
45.0           24
54.0           23
46.0           20
53.0           20
43.0           19
41.0           18
55.0           18
48.0           17
62.0           16
57.0           16
56.0           15
51.0           15
52.0           15
49.0           14
67.0           14
50.0           14
61.0           13
59.0           12
65.0           11
75.0           11
58.0           10
60.0           10
63.0           10
79.0           10
66.0           10
81.0            9
78.0            9
69.0            9
68.0            9
70.0            8
71.0            8
72.0            8
74.0            8
64.0            8
92.0            7
73.0            7
83.0            7
100.0           6
88.0            6
99.0            6
91.0            5
112.0           5
77.0            5
103.0           5
101.0           5
97.0            5
108.0           5
76.0            5
118.0           5
131.0           4
87.0            4
94.0            4
93.0            4
85.0            4
98.0            4
110.0           4
90.0            4
80.0            4
121.0           4
82.0            3
182.0           3
89.0            3
96.0            3
134.0           3
84.0            3
107.0           3
123.0           3
86.0            3
136.0           3
173.0           3
128.0           3
140.0           2
122.0           2
169.0           2
119.0           2
137.0           2
127.0           2
178.0           2
160.0           2
175.0           2
167.0           2
115.0           2
152.0           2
139.0           2
95.0            2
109.0           2
216.0           2
117.0           2
105.0           2
153.0           2
163.0           2
114.0           2
116.0           2
542.0           1
775.0           1
191.0           1
26063.0         1
188.0           1
193.0           1
176.0           1
132.0           1
133.0           1
253.0           1
189.0           1
102.0           1
202.0           1
310.0           1
4304.0          1
273.0           1
135.0           1
364.0           1
146.0           1
130.0           1
245.0           1
150.0           1
249.0           1
155.0           1
156.0           1
142.0           1
592.0           1
161.0           1
165.0           1
181.0           1
129.0           1
412.0           1
104.0           1
171.0           1
113.0           1
239.0           1
223.0           1
384.0           1
177.0           1
483.0           1
272.0           1
124.0           1
141.0           1
180.0           1
120.0           1
432.0           1
570.0           1
126.0           1
138.0           1
209.0           1
585.0           1
147.0           1
194.0           1
255.0           1
125.0           1
144.0           1
593.0           1
218.0           1
204.0           1
371.0           1
162.0           1
148.0           1
159.0           1

each value in the feature does represent a behavior for the users such that a user with 0.0 would be a low engaged user and user with 159 would be high engaged user. Each value represent the number of times a user has used the feature.

Now I am not sure what would be the correct way to deal with these kind of features. Do I do a normal transformation for these variables such log transformation such that they could be normally distributed, or should I go ahead and do binning for all these features. I am not sure if binning all the features would result in the loss of information. Or can I use the variable as it is. In using the variables as it is , I fear that model might not giving correct predictions as the low frequency values would either be in test set or train data set.

I was hoping if someone could share some thoughts/suggestions around this problem, that would be really helpful as it is very confusing for me as to what to do.

Thanks a lot in advance !!

$\endgroup$
  • $\begingroup$ What is the goal here? What is your research question? $\endgroup$ – user2974951 Oct 14 '19 at 11:21
  • $\begingroup$ @user2974951 I am working on a binary classification model where i need to predict prospective converters. Once trained the model would be deployed in production would do a weekly prediction. The question here is to understand would a feature with that kind of distribution will be an issue for the model, if so should I do binning or transformation for the variable. $\endgroup$ – Tushar Mehta Oct 14 '19 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.