0
$\begingroup$

I'm trying to reduce the size of my dataset, which is composed of 200,000 projects. Each project is defined by its size and a binary value that is 1 if the project has active users, 0 otherwise. Most of the project sizes are < 1000 (80% of the dataset).

I would like to group the projects, and for each group I would count the number of projects with active users.

I'm not sure how to group these projects. In particular, is it better to group the projects into categories with equal intervals (0-500, 501-1000, 1001-1500, ...) or into categories with the same number of projects (as a consequence, I would have unequal intervals like 0-10, 11-15, 16-40, 41-100, 101-120, ...)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I would leave the data as they are. 200,000 is not an enormous data size for anyone, unless the problem is that they need a decent computer. Not changing the dataset keeps all the detail and stops you discarding information that might be interesting or useful.

That still leaves the question of appropriate interval size for those specific occasions when a reduction is needed for display, as say for histograms or tabulations. Irregular size intervals are just puzzling to many readers (especially those better informed technically) and raise the question of what is a property of the data and what a side-effect of arbitrary choices.

A compromise could be to use intervals that are (at least approximately) equal-sized on a logarithmic scale, e.g. 1, 2, 3-4, 5-8, 9-16, etc. or possibly equal-sized on a square root scale. Logarithms and square roots are the most useful transformations for highly skewed data such as these.

(Detail: I remain unclear where project size is number of users, or something else.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for answering, I'll try to one of the transformations you suggest. The project size is the number of KBs of a project $\endgroup$
    – user2314405
    Commented Apr 9, 2014 at 11:16
  • $\begingroup$ $\lfloor \log_k \text{size}\rfloor$ will give you classes in one step, for some $k$, $\endgroup$
    – Nick Cox
    Commented Apr 9, 2014 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.