7
$\begingroup$

When in the workplace, certain measurement-taking devices are subject to different numerical accuracy; in some cases, the accuracy can be pretty weak (i.e., to one or two significant values only). Thus, instead of data sets like this: $$\{0.012, 0.033, 0.042, 0.982, 1.028, 1.037, 1.950\},$$ where each of the values are unique, we end up with a data set that looks like this: $$\{ 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 2.0\}.$$ Graphed on an individual moving range plot, the latter set appears more "chunky," and the underlying problem is estimating the true variation within the data becomes more difficult when the measurement increments are too large.

My question is the following: If I wanted a computer to detect chunky data, then I must provide a logical definition for the phenomenon. I've seen definitions that say "3 or less different values" or "4 or less different values," but I have no idea how those definitions were obtained, and what the basis/justification is for such standards.

Would anyone be able to help direct me toward a rigorous definition and justification?

$\endgroup$
5
  • 1
    $\begingroup$ For a start, what are you going to do differently according to whether the data are 'chunky' or not? $\endgroup$ – Scortchi - Reinstate Monica Feb 11 '16 at 23:06
  • $\begingroup$ I would call that data sparse, not chunky. $\endgroup$ – Vladislavs Dovgalecs Feb 11 '16 at 23:43
  • 3
    $\begingroup$ @xeon: sparse typically means "lots of zeros". While 3/7 of the values are 0, I don't think that's the aspect they are referring to. $\endgroup$ – Cliff AB Feb 11 '16 at 23:49
  • $\begingroup$ @CliffAB Extent of sparsity usually can be controlled e.g. Logistic Regression with l1-norm. In the example it looks like the values close to 0 are clamped to exact 0. $\endgroup$ – Vladislavs Dovgalecs Feb 11 '16 at 23:50
  • 3
    $\begingroup$ @xeon: with logistic regression, you can increase the sparsity of the estimated effects with Lasso, as you mentioned. But the OP is asking about the data being rounded, which may mean the values are sparse if many of them fall in the interval [-0.5, 0.5), for example, but certainly not necessarily. Again, I don't think they are asking "what do I do if a lot of my data is equal to 0?", but rather "what if I have binned data"? $\endgroup$ – Cliff AB Feb 11 '16 at 23:58
8
$\begingroup$

Data like this is often called quantized, particularly when the numbers' precision is limited by the measurement device. For example, a scale might only display integer numbers of grams or pounds. This is particularly common when an analog signal (from a microphone, strain gauge, etc) is digitized. The resulting error (e.g., the difference between 0.012 and 0 for your first data point) is called quantization error. You could also call it rounding or discretization, though this faintly implies that it was done during post-processing.

Truncation also works here, but one needs to distinguish between truncating the range of the observations (e.g., converting anything above 10 into 10, or below 0 to 0) and truncating the values of individual observations.


I'm not aware of a way to robustly detect quantization in any situation. In fact, pretty much all data is quantized to some extent and the amount of quanitization is often known ahead of time from the measuring device's specifications. However, there are some easy heuristics you could try:

  • How many unique values do you have? Digital-to-analog converters use a fixed number of bits (typically 8, 12, 16, or 24), which gives you $2^8, 2^{12}, 2^{16}$ or $2^{24}$ unique values, and these values are often equally spaced between the maximum and minimum value.

  • Is there a consistent step-size between the values. In other words, sort them, throw out duplicates, and see if the neighboring values typically increase by the same amount.

Still, I think you'd be better off inquiring about how the data was generated to begin with.


If the data is "mildly" quantized, it's usually not an issue. For example, I wouldn't worry too much if my human subjects' weights were recorded in (integer) pounds or kilograms. If the data is heavily quantized, you could treat it as interval-censored data. This is particularly common in survival analyses, where you might only check to see if someone is alive or something is functioning at some fixed interval (e.g., weekly inspections of a factory). Search for interval regression if this fits your situation.

You should be sure to understand the null hypothesis underlying any tests you run on binned data. For example, data uniformly distributed across 10 bins is quite different from data uniformly distributed across the entire range.

$\endgroup$
6
$\begingroup$

Generally, "binned data" is how this is referred to.

If you think of a histogram, each bar refers to a bin. If a value is between the upper and lower ends of a given bin, that value is placed in this bin. As an example, if you have binned data due to simple rounding (i.e. a true value of 1.01 becomes represented as 1.0 in the dataset), you can think of observed value 1.0 meaning the true value was actually in the interval [0.5, 1.5).

Typically, this aspect of the data is often ignored; there's often little issue with using the integer age of a subject (28 years) rather than the exact age (28.153...). In the cases the binning effect may be substantial (i.e. years at company; 1/12 is much different than 5/12, but rounded they are both 0), the data can be treated as interval censored to account for this uncertainty in exact response value.

$\endgroup$
2
$\begingroup$

In your case it is called quantization, a common issue with signal processing. Typically you see evenly spaced data (even when you get no multiplicities).

In general (is there are many points close to each other, not necessary with the same value or spacing), look at clustering. For a 1-dimensional values sort them and take a histogram of differences between nearest values.

$\endgroup$
0
$\begingroup$

To add to the other good answers, and more of a comment on the sources of the chunkiness - quantization can occur for social reasons too, e.g. if you look at a histogram of the diamond dataset in detail you will see pronounced spikes at "nice" values, 0.3, 0.4, 0.5, 0.7, 1.0, 1.2, 1.5, 2.0 etc. There are very few diamonds of weight 0.98, but lots of with a weight just over 1.0, which was explained as - nobody wants to be given a 0.98 carat diamond ... they want a 1.0 carat diamond !!

enter image description here

library(ggplot2)
data(diamonds)
ggplot(diamonds, aes(x=carat)) + geom_histogram(bins=200) + xlim(0,2.1)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.