I have a set of data that is not given as $ \boldsymbol{x} = x_1, \dots, x_n,$ but as pairs $\boldsymbol{x}_{interval} = (x^{(start)}_1, x^{(end)}_1), \dots, (x^{(start)}_n, x^{(end)}_n). $ For each pair $(x^{(start)}_i, x^{(end)}_i),$ the true $x_i$ lies in the interval $(x^{(start)}_i, x^{(end)}_i), $ but it is not known where.

In context, this means that we have intervals in which we know an event $x_i$ occured, which tells us that it happened after $x^{(start)}_i,$ but before $x^{(end)}_i$.

The goal of the analysis is to model this data or in some way approximate the distribution. Initially, I'll start with trying to use the information contained in the intervals to fit a normal distribution to the distribution of the unobserved events $\boldsymbol{x}$ .

I'm having a very hard time finding any information about this type of problem. Is this a known field of research, of statistical interval analysis?

  • 1
    $\begingroup$ It is a little like double sided censoring in survival analysis. But it is not a time censoring and you have several intervals with only one observation per interval. If I were to put a distribution for the unknown x in each interval I would use the uniform distribution because I think you have no reason to favor any particular points in the intervals over others. $\endgroup$ – Michael R. Chernick Jan 12 '17 at 14:50
  • 2
    $\begingroup$ There is a tag on this site for interval-censoring, perhaps some of the answers there may help you? $\endgroup$ – mdewey Jan 12 '17 at 14:59
  • 1
    $\begingroup$ It is conceivable that how the intervals are generated could be informative. As such, I would be reluctant to apply some omnibus "interval-censoring" technique to the analysis of these data until I had a better understanding of what these intervals actually represent. Could you elaborate on that? For instance, many digital meters truncate measurements to some nearest power of 10, whence all intervals have a common width with predefined endpoints. In other cases the interval widths might vary with the measurement due to properties of the measurement itself. $\endgroup$ – whuber Jan 12 '17 at 15:10

The data are censored, specifically interval-censored. Censoring, especially right-censoring (start but no end), is a common feature of time-to-event data and dealt with under survival analysis (Medicine) or reliability analysis (Engineering).

For parametric modelling of such data the key insight is that contributions to the joint likelihood from uncensored data are of the form $$f(x_i)$$ while those from censored data are of the form $$F\left(x_i^\mathrm{(end)}\right)-F\left(x_i^\mathrm{(start)}\right),$$ where $f(\cdot)$ is the density & $F(\cdot)$ the distribution function. Under the assumption of independent censoring—which you shouldn't leap to—these are the only part of the likelihood needed for inference as the censoring times contain no additional information about the parameters. If a normal distribution seems appropriate start off with a contour plot of the likelihood against the mean & variance parameters, then improve initial maximum-likelihood estimates numerically.

  • $\begingroup$ Thanks so much for the connection to the term censoring, and interval-censoring. The likelihood contribution as a difference between the two CDFs was indeed my hunch, but it's great to see it makes sense. Thanks again! $\endgroup$ – Kees Mulder Jan 13 '17 at 10:47

A good starting for examining the univariate distribution would be to look at the Non-Parametric Maximum Likelihood Estimator (NPMLE). This is a generalization of the Kaplan-Meier curves (which itself is a generalization of the Empirical Distribution Function), which will give you a non-parametric estimate of the cumulative distribution function. Interestingly, this estimate is not unique (unlike the EDF or Kaplan Meier curves), but rather known up to an interval. So you will get a pair of step functions that bound the NPMLE, rather than a single step function.

While this estimator is good for examining the shape of a distribution, it can be a bit unstable, i.e. high variance in the estimates. One can fit standard parametric models, but it is still recommended to use the NPMLE at least for model checking.

Many of the standard survival regression models are available (proportional hazards, accelerated failure time and proportional odds, for example). Interestingly, although the NPMLE has high variance for the estimates of the survival curve, the regression parameters in a semi-parametric model which uses the NPMLE for the baseline distrubtion do not suffer from the instability. So semi-parametric regression methods are quite popular for inference.

@Scortchi and @whuber bring up important points about the generation of the beginning and end of the observation intervals ($x_i^{start}, x_i^{end}$ as defined by the OP). A standard simplifying assumption (which should be carefully considered) is that there are a set of inspection times $C_0 \leq C_1 \leq, ..., \leq C_k$ that are generated independently of the actual event time / outcome $t$ of interest (equality occurs when we observe the event time exactly). Then, all we observe is the interval $C_j, C_{j+1}$ such that $t \in C_j, C_{j+1}$. But if it seems plausible that the event time could strongly influence inspection time, care must be taken in the analysis. As an example, suppose our event of interest was onset of tooth decay and our inspections were dentist visits. If we go to the dentist fairly regularly, then the assumption of independence seems reasonable. But if we very rarely go to the dentist except when our tooth hurts a lot, then $t$ is definitely influencing $C_j$!

A brief tutorial for using these models in my R-package icenReg can be found here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.